Indexing time issue after upgrading 7.7.1 to 7.10.1

Hello everyone!

Thanks for your product!
We have several Elasticsearch clusters and one of them has a problem when upgrading from 7.7.1 to 7.10.1.
We needed an update due to a problem with the circuit breaker (https://github.com/elastic/elasticsearch/pull/59394)

We use Elasticsearch as a database for searching and displaying data.
Our environment:

  • Elasticsearch is running in docker (in k8s), image docker.elastic.co/elasticsearch/elasticsearch-oss
  • Exporter metrics justwatch/elasticsearch_exporter:1.1.0
  • Load Profile: ~ 2000-3000 bulk (batch 200) inserts data at each node and 150-200 read requests to each node.
  • 3 servers (8 CPU (Xeon E3-1240 v6 3.7Ghz) / 64 Gb RAM / 4x480 Gb SSD (JBOD)).
  • 1 server (16 CPU (Xeon E-2288G 3.7Ghz) / 64 Gb RAM / 4x480 Gb SSD (JBOD)).
  • POD limits: memory 52Gi && cpu 8
  • heap: -Xmx32127m -Xms32127m
  • 1 main index (most active): 80 Gb for 3 shards. 2 replicas for each shard.

Main problem:

  • After updating ES to 7.10.1, indexing time increased several times, which led to product problems (we do not have time to update data in elasticsearch) and almost all server resources were consumed.

This is how the cpu load looks like:

From the application side, nothing has changed. The application profile with ES has not changed.
For the last week we have been investigating the problem and cannot find a solution to this question: so that the indexing time returns to its previous values.

The only thing we noticed - it is a serious change of memory used for index segments:

Displayed metric: elasticsearch_indices_segments_memory_bytes
But we did not find how we can influence this metric in order to try to return it to its old values or to see the influence of its changes on it.


Please help in the analysis of this problem.

  • How can we affect elasticsearch_indices_segments_memory_bytes?
  • Maybe we needed to bring specific changes to elasticsearch after upgrading to version 7.10.1?
  • Maybe we need to provide more information on our configuration / load to understand the problem?

Thank you.

@warkolm hi!

Can you help me with this topic?
Or suggest - can anyone help me?

This should be set to around 30GB in order to benefit from compressed pointers. Please see this blog post for further details. It is recommended that the heap is set no larger than 50% of available RAM, which in your case would be 26GB.

@Christian_Dahlqvist, thanks for the suggestion!

Compressed pointers work, but improvements are not visible - the use of the CPU and the indexing time did not change :frowning:

Nobody has such problems on the latest versions of Elasticsearch?