Degraded Indexing Performance on v7.3.1 (from v5.6.10)

Hey,

We recently upgraded from ES version 5.6.10 to 7.3.1 and noticed indexing performance seems much worse. We could index at a rate of around 500k op/s, and now we peak at around 200k op/s.

During indexing, we get bulk request thread rejections, but we are actually only utilising around 50% of our available CPU, and it seems we are not able to increase the thread pool size because of this limit here: https://github.com/elastic/elasticsearch/blob/master/server/src/main/java/org/elasticsearch/threadpool/ExecutorBuilder.java#L52

Our use case is that we only reindex clusters that don't receive live traffic, so I don't think this limit really makes sense for us. I wondered what the reasoning behind it was, and if other people have the same issue? I assume if we could increase the thread pool size, we would be able to utilise ~100% of CPU for indexing.

Steps to reproduce :

  1. Reindex a cluster with enough writes per second to get bulk thread rejections
  2. Check if CPU is actually maxing out
  3. In our case, we're only using 50% CPU

Cheers!
Jack

Are you indexing new documents only or do you also update existing documents? What does disk I/O and iowait look like? How many indices and shards are you actively indexing into?

  1. We are indexing all new documents
  2. Disk IO utilisation is around 10%, and CPU wait IO is <0.3 (the machines have 32 cores)
  3. We're indexing into 15 primary shards and it's all 1 single index

And in this case we're using about 50% CPU of the cluster. But increasing the number of bulk requests will start to cause thread rejections.

Why didn't you update to the latest version? At least to 7.5

7.3.1 was the latest version when we upgraded :slight_smile:

Besides, this limitation is still there in recent versions.

I see that it's reading node.processors setting to configure the thread pool limit...is it possible to override the number of available processors in the elasticsearch config somehow?

Ah found it, the parameter is just processors in 7.3.1
Seems a bit hacky to have to fake the number of available processors to increase this limit, but hey ho.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.