# Understanding elasticsearch filter context **URL:** \ **Category:** Elasticsearch\ **Tags:** language-clients\ **Created:** [February 4, 2022, 5:30am UTC](https://discuss.elastic.co/t/understanding-elasticsearch-filter-context/296250 "2022-02-04T05:30:21Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Dheeraj\_Gupta](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@Dheeraj\_Gupta](https://discuss.elastic.co/u/Dheeraj_Gupta)\ **Post date:** [February 4, 2022, 5:30am UTC](https://discuss.elastic.co/t/understanding-elasticsearch-filter-context/296250/1 "2022-02-04T05:30:21Z")
I have some doubts about how the filter context works in Elasticsearch/Lucene ecosystem which I would be grateful if someone can clear up. This is related to [another problem](https://discuss.elastic.co/t/very-slow-search-performance-after-upgrade-to-7-16-1/296152/2) which I have been recently facing. I have an index that stores DNS request/responses. The index is per day and older indices are force merged to 1 segment. Each daily index has around 50-60M docs (total 12-15G) divided in to 2 shards with replica set to 1. Typical documents have following schema ```auto {"source_ip": IP, "destination_ip": IP, "source_port": long, "destination_port": long, "sensor": keyword, "query": keyword, "answers": text, "@timestamp", ... } ``` Most of entries have same `destination_ip` while rest of the fields are different. The `destination_ip` breakup is - `10.11.100.100` - 50% - `10.138.100.100` - 35% - Others - 15% Typically I need to search for (`source_ip`, `destination_ip`, `query`, `sensor`) within a time window and only return documents that match ALL these criteria. This search is against older indices (that have been force-merged). From the docs I understand `bool` query with `filter` is the best bet in such cases as that uses the filter context. Using `elasticsearch-py and elasticsearch-dsl`, I usually form queries like so ```auto src_srch = (ESD.Search(using=es, index="dns-YYYY.MM.DD") .filter("range", **{"@timestamp": {"lte": lseen, "gte": fseen}} .filter("terms", **{"source_ip": ["xxxx"]}) .filter("terms", **{"destination_ip": ["xxxx"]}) .filter("terms", **{"sensor": ["xxxx"]}) .filter("terms", **{"query": ["xxxx"]})) ``` This gets translated to `bool` with multiple `filters`. Does Elasticsearch do some query optimization on filters or does it pass this responsibility to Lucene? How does Lucene do this? Are the filters run in a particular order? Filter context caches the results so for above query will separate cache be created for each filter or single cache with aggregate results of all the filters? The problem that I am hitting is that after upgrade to 7.16.1 the queries have become very slow (As shown in linked issue). Profiling the query of type: `{"source_ip": XX, "destination_ip": "10.11.100.100", "query": YY, "sensors": ZZ} in time period (a, b)` shows query takes 300ms on one shard (hits \< 10) with bulk of time spent in filtering `destination_ip:10.11.100.100`. If I trim the query to remove `destination_ip` `{"source_ip": XX, "query": YY, "sensors": ZZ} in time period (a, b)`, profiler shows same query completing in \< 1ms (Same number of hits) as most expensive filter is not run. Are there any particular reasons this could be happening? ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [February 8, 2022, 1:51pm UTC](https://discuss.elastic.co/t/understanding-elasticsearch-filter-context/296250/2 "2022-02-08T13:51:44Z")
Hey, so filters in a bool query sound like the most efficient idea here, as long as you don't need any scoring. What happens internally is, that Lucene is able to cache each filter separately, if it is being reused a certain number of times. See also [Elasticsearch caching deep dive: Boosting query speed one cache at a time | Elastic Blog](https://www.elastic.co/blog/elasticsearch-caching-deep-dive-boosting-query-speed-one-cache-at-a-time) That said, I suppose it makes more sense to focus on the issue in the other post you wrote first, see if that can get fixed and then go from there. --Alex ---
**Author:** ![Dheeraj\_Gupta](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@Dheeraj\_Gupta](https://discuss.elastic.co/u/Dheeraj_Gupta)\ **Post date:** [February 8, 2022, 2:18pm UTC](https://discuss.elastic.co/t/understanding-elasticsearch-filter-context/296250/3 "2022-02-08T14:18:25Z")
Thanks @spinscale for the explanation regarding Lucene caching! ---
**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\ **Post date:** [February 8, 2022, 3:23pm UTC](https://discuss.elastic.co/t/understanding-elasticsearch-filter-context/296250/4 "2022-02-08T15:23:49Z")
> [@Dheeraj\_Gupta](#): > > Does Elasticsearch do some query optimization on filters or does it pass this responsibility to Lucene? How does Lucene do this? Are the filters run in a particular order? I found a post explaining some of these questions. It's not the recent one, but I suspect the general framework is still the same. > **[In which order are my Elasticsearch queries/filters executed?](https://www.elastic.co/blog/elasticsearch-query-execution-order)** And also, though I'm not sure it will mitigate the problem, [Index Sorting](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-index-sorting.html) looks related to the problem. ---
**Author:** ![Dheeraj\_Gupta](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@Dheeraj\_Gupta](https://discuss.elastic.co/u/Dheeraj_Gupta)\ **Post date:** [February 9, 2022, 6:52am UTC](https://discuss.elastic.co/t/understanding-elasticsearch-filter-context/296250/5 "2022-02-09T06:52:04Z")
Thanks, I had seen this link but was hoping for something more recent. Index sorting unfortunately will not fit the bill as the indexes are write heavy. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 9, 2022, 6:52am UTC](https://discuss.elastic.co/t/understanding-elasticsearch-filter-context/296250/6 "2022-03-09T06:52:34Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.