Indexing stopped during field data cache clear

Elastic Stack Elasticsearch
1

Hi,

We observed that Indexing stopped on our ES cluster while we were running the curl command to clear up the field data cache. The field data cache was using ~1 GB on each node(54 data node cluster) while the total heap allocated to ES is 30G. Clearing up the cache took us around 2 minutes and indexing stopped for 2 minutes.

I'm unable to understand how clearing up the cache caused indexing to stop? I'm more concerned because this happened on our production cluster, and would like to know if there are any configurations that must be set before clearing the cache to prevent this in future.

2

It might have been GC at the same time, is there anything in the logs?

Why did you clear it?

3

Thanks Mark, yes it was GC. Looked through logs and found that the nodes were going through old gc at the same time taking almost 15-18sec. That's a long time for a service to stop responding. We needed to clear the cache as we use doc_values for sorting/aggregation, a random score query somewhere was using the field cache which we disabled, so it was better to reclaim the memory back from field data cache.