Skip to content

Optimize gstore initialization

Xiating Xie requested to merge xiext/wukong:store_init into master

Optimize gstore loading, specifically remove extra one time memory overhead of triples_map in init_triples_map.

This triples_map is used to group all the data of each segments and then accelerate triples insert. It copies all the triples data for one time and consumes much memory.

Instead, I use a hashtable to store only the start offset of each segments.

Test:

  • lubm_2560(lubm_q1 - lubm_q12), lubm_640_attr (lubm_attr_q1 - lubm_attr_q5)
  • DUSE_DYNAMIC_GSTORE = ON/OFF, DUSE_VERSATILE = ON / OFF
  • Under all above settings, all the query tests passed.

Evaluation: lubm_2560 dataset, one machines, 16 engines, DUSE_VERSATILE=ON

  • Original: 260 secs
  • Optimized: 160 secs
  • Optimized wukong can run lubm_10240 on two machines with 100GB available memory.

Merge request reports