Elasticsearch poor indexing performance

[apyshchyk (Andriy)]

Hi everyone,
I have issues with ES indexing performance.
First of all - current cluster setup 15 data nodes 8CPU, 32RAM
3 Master nodes 4CPU, 16 RAM
external monitoring cluster.
For current performance testing I'm sending 25K events/s, which is distributed by multiple indices (totally to ~ 600shards)
sometimes I see huge performance degradation, looking into _cat/thread_pool/bulk I have a response like this

There is 10 threads of indexing applications, which hitting multiple indices during bulk insert
Looks like only 1 data node is busy during this time, others 14 - doing nothing. after a while - speed get back to normal. How to prevent this situation?

[Christian_Dahlqvist]

Which Elasticsearch version are you using? What is the size of your documents? What indexing throughput are you seeing? What size bulk requests are you using?

[apyshchyk (Andriy)]

Forgot to mention,
ES Version 5.5.0
indexing throughput when all nodes are loaded ~ 20-25K event/s
indexing throughput when only one got huge bulk queue - 3-5K events/s
IDocument size ~1Kb
Bulk size - 10Mb

[Christian_Dahlqvist]

What is the ratio between inserts of new data and updates to existing documents? Are you using parent-child or nested documents?

[apyshchyk (Andriy)]

~5 updates per 5 Millions documents (when message accidently sent twice)
No any parent-child or nested documents,

[apyshchyk (Andriy)]

Only indexing is happening in ES Cluster now, but speed is fluctuating a lot (speed is total for primary and replica, to have real indexing speed, divide by 2)