JVM heap usage issue due to percolate queries

[Anatoly_Petkevich]

Hi,
Document percolation in our environment periodically causes high JVM usage with a steady grow and reaches max memory limit.
We cannot figure out what causes this issue, but it looks like as a memory leak related to dandling in-memory temporary Lucene indices used by percolator.
Many input document have nested fields.

ES version is 5.2.2.
Thank you.

[thiago (Thiago Souza)]

Hello Anatoly,

In 5.x, the class org.elasticsearch.index.cache.bitset.BitsetFilterCache is actually the node query cache. Indeed ~75% is too high (default is 10%).

Check if your elasticsearch.yml sets indices.queries.cache.size.

Regards

[nimes]

Thiago,

We don't set indices.queries.cache.size, leaving it by default.
The issue affects only nodes having percolate indices (see heap dump from one of those nodes in Anatoly's post above).
The heap dump below is from one of the healthy nodes in the same cluster that don't have percolate indices:

Thank you

[thiago (Thiago Souza)]

Hello,

Please attach your complete elasticsearch.yml

Cheers

[nimes]

Thiago,

See below elasticsearch.yml from our test cluster where the issue is reproduced (some settings not related to the issue were omitted, like discovery, xpack etc.):

bootstrap.memory_lock: true
indices.memory.index_buffer_size: 30%
indices.memory.min_index_buffer_size: 96mb
thread_pool.bulk.size: '3'
thread_pool.bulk.queue_size: "-1"

Thanks

[Anatoly_Petkevich]

Hello,
Could be this issue related to [Percolator] Remove caching and support "now" in range queries ?

[Anatoly_Petkevich]

A bug https://github.com/elastic/elasticsearch/issues/24108 has been created