ES bulk index performance

Elastic Stack Elasticsearch
1

Hello,

We are using ES 7.2.0 to index some documents. Using node.js we convert csv documents to json and use bulk index with 5000 batch size.

We have two different files with different number of fields. All field types are long or double. We use dynamic mapping.

For the file who has 40 fields, it takes 30seconds to index 340.000 documents. Kibana shows index rate 10000/s

For the file who has 273 fields it takes 4 minutes to index 340.000 documents. Kibana shows index rate 1250/s.

Is this performance normal? Why number of fields affect the performance too much?

I tried recommended settings to improve performance but didnt work. How can i get better index rate?

System Properties & Configuration

  • one node

  • Intel(R) Xeon(R) CPU E5-1620 v4 @3.50 Ghz
    -32GB RAM
    -8 processors

  • bootstrap.memory_lock: true
    -indices.memory.index_buffer_size:"1g"
    -refresh_interval:15s

-Xms8g
-Xmx8g

Thank you.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.