Elastic indexing rate decreased after a while


(Ghasem) #1

I have a problem with bulk indexing rate. As you can see in following graph, after a while the ram usage starts to increse and the indexing rate start to decrese rapidly.
Why?

Server specification:
CPU: Core i7
RAM: 12 g

Elastic Configuration:
Bulk size: 30,000
ES_Heap_Size:5g
Nodes:1
Number of shards: 50
Number of replica: 0
bootstrap.mlockall: true
indices.merge.scheduler.max_thread_count: 1
indices.translog.flush_threshold_size : 1G
transient.indices.store.throttle.type : none
index. compound_on_flush: false,
index. compound_format: false,
index. Warmer.enable: false,
index.refresh_interval: -1,

Data specification:
Number of records: 20,000,000


(Amin) #2

Dear Ghasem,

As you can see in the attached picture, when a merge process starts, the ram usage start to increase too.
So, the indexing rate decreases rapidly.
you have to have a look at this link for more details.


(system) #3