Indexing Speed Degrade With the Time

Elastic Stack Elasticsearch
1

Hi,
I am doing some performance tuning in elastic search for my project and I need some help in improving the elastic search indexing speed. I am using ES 5.1.1 and I have 2 nodes setup with 8 shards for the index. I have the servers for 2 nodes with 16GB RAM and 12CPUs allocated for each server with 2.2GHz clock speed. I need to index around 25,000,000 documents within 1.5 hours, which I am currently doing in around 4 hours. I have done the following config changes to improve the indexing time.

  1. Setting 'indices.store.throttle.type' to 'none'
  2. Setting 'refresh_interval' to '-1'
  3. Increasing 'translog.flush_threshold_size' to 1GB
  4. Setting 'number_of_replicas' to '0'
  5. Using 8 shards for the index
  6. Setting VM Options -Xms8g -Xmx8g (Half of the RAM size)

I am using the bulk processor to generate the documents in my java application and I'm using the following configurations to setup the bulk processor.

Bulk Actions Count : 10000
Bulk Size in MB : 100
Concurrent Requests : 100
Flush Interval : 30

Initially I can index around 356167 in the first minute. But with the time, It decreases and after around 1 hour its around 121280 docs per minute.

How can I keep the indexing rate steady over the time? Is there any other ways to improve the performance?

Thanks in advance.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.