# Stream indexing and querying filter caching **URL:** \ **Category:** Elasticsearch\ **Created:** [October 7, 2015, 11:51am UTC](https://discuss.elastic.co/t/stream-indexing-and-querying-filter-caching/31768 "2015-10-07T11:51:10Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Ilija\_Subasic](https://avatars.discourse-cdn.com/v4/letter/i/ee7513/32.png) [@Ilija\_Subasic](https://discuss.elastic.co/u/Ilija_Subasic)\ **Post date:** [October 7, 2015, 11:51am UTC](https://discuss.elastic.co/t/stream-indexing-and-querying-filter-caching/31768/1 "2015-10-07T11:51:10Z")
Hi, We have a system which does stream indexing for an online system that has a large number of filtered queries. As filter scope changes it looks like there is a lot of filter evictions going on. I have a couple of questions if anyone can help: a) is filter rebuild with each newly indexed file that fits into the scope? does this happen on querey time? b) what is the order of filter eviction (e.g. FIFO) ? c) does anyone have experience with preformance with manually contorling some filter? d) what is the cost of continiously adding documents to filters (e.g. imagine that I have a field "document\_type" filter and that every second I index 100 new documents for a type)? Any help would be highly welcomed. I browsed a code a bit, but am not 100% sure I get everything about filter caching and building. Best, Ilija ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 8, 2015, 7:48am UTC](https://discuss.elastic.co/t/stream-indexing-and-querying-filter-caching/31768/2 "2015-10-08T07:48:12Z")
> [@Ilija\_Subasic](#): > > a) is filter rebuild with each newly indexed file that fits into the scope? does this happen on querey time? Only when a segment is created/merged. > [@Ilija\_Subasic](#): > > b) what is the order of filter eviction (e.g. FIFO) ? Last recently used. > [@Ilija\_Subasic](#): > > c) does anyone have experience with preformance with manually contorling some filter? You can tune some things but you need to be careful as you can end up causing more issues. Can you increase heap? > [@Ilija\_Subasic](#): > > d) what is the cost of continiously adding documents to filters (e.g. imagine that I have a field "document\_type" filter and that every second I index 100 new documents for a type)? See 1; you probably wouldn't notice it under normal loads though. ---
**Author:** ![Ilija\_Subasic](https://avatars.discourse-cdn.com/v4/letter/i/ee7513/32.png) [@Ilija\_Subasic](https://discuss.elastic.co/u/Ilija_Subasic)\ **Post date:** [October 9, 2015, 9:07am UTC](https://discuss.elastic.co/t/stream-indexing-and-querying-filter-caching/31768/3 "2015-10-09T09:07:00Z")
Thanks for the answer. > [@warkolm](#): > > You can tune some things but you need to be careful as you can end up causing more issues. Can you increase heap? What is the worst that can happen, how much does it cost to build an index? > [@warkolm](#): > > You can tune some things but you need to be careful as you can end up causing more issues. Can you increase heap? We are already at 32 GB, so guess we could add more nodes. But most filters are used only once (e.g. time filters from the query moment to some predefined). > [@warkolm](#): > > See 1; you probably wouldn't notice it under normal loads though. We have waves of indexing 10k+ a second, normally it is in 100s. Plus each indexing event is followed by a number of queries. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:45pm UTC](https://discuss.elastic.co/t/stream-indexing-and-querying-filter-caching/31768/4 "2017-07-05T23:45:42Z")