Lots of filter cache evictions in Elasticsearch when used with Kibana

Elastic Stack Elasticsearch
1

Hi there!

I'm seeing lot's of filter cache evictions in Elasticsearch 1.7.1 when I use some more complex Dashboards in Kibana 4.1.1. When I restart the whole cluster (it's a testing environment) it takes about 4 hours until the filter cache reaches the maximum I configured in elasticsearch.yml .

I wonder why this happens because I only use the same two dashboards (lots's of visualisations with terms and some line charts with date histogram. I only change the timeframe I want to see (from last 12 hours, to this day, to this week, to last 7 days) for testing.

I raised the percentage of filter cache and expected a positive performance impact. I did not yet measure response times because there are some other issues in the infrastructure to fix but at least I did not expect the filter cache to fill up start evicting.

In numbers, there are about 8 evictions per query!

I checked with some other ELK instances I can connect to and they keep at 0.7 evictions per query even after 14 days.

So am I just overreacting or is there something wrong?

Cheers,
Thomas

2

I'm not super familiar with the internals of how Kibana uses filters, but I believe it is basically constructing time filters on top of buckets for your selected time window.

These windows will be relatively unique and not great for cache-reuse. If you select the last seven days, your filters will basically be from: <some_time> to:<some_other_time>. If you wait an hour and refresh, the filter becomes from: <some_time + 1hr> to:<some_other_time + 1hr>. Every ms, the old filter is essentially obsoleted, unless you are looking at the exact same time slice rather than a relative slice ("last XX period").

Elasticsearch doesn't cache filters using now expressions by default, to prevent exactly this kind of churn. The problem is that (I think) Kibana provides explicit start/stop times, which means the date expression is cached, never used again and churned out.

I believe this is a related ticket: https://github.com/elastic/kibana/issues/4654

With that said, I wouldn't worry about it toooo much, especially if you aren't seeing performance problems. It should be noted that not all evictions are equal either. An eviction on a small, newly created segment is much less expensive than an eviction on a very large segment...but they look equal in the stats.

Also, with time-based data, you tend to look at a certain "hot" window (say, the last week or month), so it's natural for older filters to churn out once they exceed that window.

3

Thanks für your reply. This helps in cooling down some people always asking why "something's red in HQ plugin". :wink:

I will keep an eye on the issue and keep living with these values for now. And I will try to get more dashboard builders to use more cacheable timeframes, if possible.