What are field evictions?

Elastic Stack Elasticsearch
1

In the HQ plugin of my elasticsearch cluster, I see warnings about the frequency of field evictions.

What is a field conviction? Are they bad? How can I reduce them?

===============================================================================

===============================================================================

2

Elasticsearch caches filters and filter evictions can happen for two reasons:

  • you have a rapidly changing index and deleted "segments" trigger evictions of the cache entries that are associated to them
  • you use a lot of filters that elasticsearch can't all cache so elasticsearch will evict the least-recently used filters in order to make room for new filters

4.7 filter evictions per query is indeed quite high. You can look at your queries to see if there are filters that you do not reuse, in which case it would help to turn off caching explicitly (_cache: false).

Note: lots of improvements about filter caching are coming to 2.0 which should make this situation better.

3

Could you elaborate what a deleted segment is?

The cluster is only bringing in about 5GB per day of logs from log stash. I am the only person using the cluster right now.

I have turned on doc_values to help with a performance problem.

4

Not sure if these really filter evictions are really a concern, but sure, this served as an indication some metric is out of ordinary range. From elasticsearch user point of view, do you face queries become slow even with filter evictions happened? We have this same issue too but when we make queries, the response duration still at acceptable range.

In es version 0.90, there are node stats that expose fielddata https://www.elastic.co/guide/en/elasticsearch/reference/0.90/cluster-nodes-stats.html but with es version 1.5, afaik, there are more metric expose, from that metric output, it should give some idea about what fielddata/filter are using a lot of heap.

hth

jason

5

Clinton wrote a nice explanation of what segments are at lucene - Understanding Segments in Elasticsearch - Stack Overflow. Your data is stored into immutable segments and we rely on this for filter caching: since segments don't change the cache never needs invalidation. However updating the index sometimes requires to add new segments, merge existing segments together and remove old segments. When an old segment is removed, we just trash everything that was cached on that segment.