Elasticsearch poor indexing performance

Hi everyone,
I have issues with ES indexing performance.
First of all - current cluster setup 15 data nodes 8CPU, 32RAM
3 Master nodes 4CPU, 16 RAM
external monitoring cluster.
For current performance testing I'm sending 25K events/s, which is distributed by multiple indices (totally to ~ 600shards)
sometimes I see huge performance degradation, looking into _cat/thread_pool/bulk I have a response like this
47 PM

There is 10 threads of indexing applications, which hitting multiple indices during bulk insert
Looks like only 1 data node is busy during this time, others 14 - doing nothing. after a while - speed get back to normal. How to prevent this situation?

Which Elasticsearch version are you using? What is the size of your documents? What indexing throughput are you seeing? What size bulk requests are you using?

Forgot to mention,
ES Version 5.5.0
indexing throughput when all nodes are loaded ~ 20-25K event/s
indexing throughput when only one got huge bulk queue - 3-5K events/s
IDocument size ~1Kb
Bulk size - 10Mb

What is the ratio between inserts of new data and updates to existing documents? Are you using parent-child or nested documents?

~5 updates per 5 Millions documents (when message accidently sent twice)
No any parent-child or nested documents,

Only indexing is happening in ES Cluster now, but speed is fluctuating a lot (speed is total for primary and replica, to have real indexing speed, divide by 2)

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.