Degraded Indexing Performance on v7.3.1 (from v5.6.10)

Elastic Stack Elasticsearch
1

Hey,

We recently upgraded from ES version 5.6.10 to 7.3.1 and noticed indexing performance seems much worse. We could index at a rate of around 500k op/s, and now we peak at around 200k op/s.

During indexing, we get bulk request thread rejections, but we are actually only utilising around 50% of our available CPU, and it seems we are not able to increase the thread pool size because of this limit here: https://github.com/elastic/elasticsearch/blob/master/server/src/main/java/org/elasticsearch/threadpool/ExecutorBuilder.java#L52

Our use case is that we only reindex clusters that don't receive live traffic, so I don't think this limit really makes sense for us. I wondered what the reasoning behind it was, and if other people have the same issue? I assume if we could increase the thread pool size, we would be able to utilise ~100% of CPU for indexing.

Steps to reproduce :

  1. Reindex a cluster with enough writes per second to get bulk thread rejections
  2. Check if CPU is actually maxing out
  3. In our case, we're only using 50% CPU

Cheers!
Jack

2

Are you indexing new documents only or do you also update existing documents? What does disk I/O and iowait look like? How many indices and shards are you actively indexing into?

3
  1. We are indexing all new documents
  2. Disk IO utilisation is around 10%, and CPU wait IO is <0.3 (the machines have 32 cores)
  3. We're indexing into 15 primary shards and it's all 1 single index

And in this case we're using about 50% CPU of the cluster. But increasing the number of bulk requests will start to cause thread rejections.

4

Why didn't you update to the latest version? At least to 7.5

5

7.3.1 was the latest version when we upgraded :slight_smile:

Besides, this limitation is still there in recent versions.

I see that it's reading node.processors setting to configure the thread pool limit...is it possible to override the number of available processors in the elasticsearch config somehow?

6

Ah found it, the parameter is just processors in 7.3.1
Seems a bit hacky to have to fake the number of available processors to increase this limit, but hey ho.