# Filtering to a subset of the full index **URL:** \ **Category:** Elasticsearch\ **Created:** [July 17, 2015, 11:47am UTC](https://discuss.elastic.co/t/filtering-to-a-subset-of-the-full-index/25795 "2015-07-17T11:47:37Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Phil\_Messenger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phil_messenger/32/45627_2.png) [@Phil\_Messenger](https://discuss.elastic.co/u/Phil_Messenger)\ **Post date:** [July 17, 2015, 11:47am UTC](https://discuss.elastic.co/t/filtering-to-a-subset-of-the-full-index/25795/1 "2015-07-17T11:47:37Z")
I have a large (tens of billions of docs) index spread across a bunch of machines. I'd like to constrain a query to a subset of that index. Example: each doc might be a message with an author, and I want to constrain a search to messages written by a subset of all authors. The catch is that this subset may still be a million people, so clearly a boolean query isn't going to work. I'm wondering if I can do this with a custom plugin. I could probably do something with a custom scorer, but that doesn't feel very efficient. I'd need to keep a cache of docId -\> authourId in memory for this to work at a reasonable speed. Am I missing something obvious? ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [July 17, 2015, 12:06pm UTC](https://discuss.elastic.co/t/filtering-to-a-subset-of-the-full-index/25795/2 "2015-07-17T12:06:03Z")
Bool queries should work fine. Just stick a term filter in the should clause. You can limit the fanout with routing I'd all your authors are small. That can create hotspots if some are large but you can work around that in other ways too. ---
**Author:** ![Phil\_Messenger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phil_messenger/32/45627_2.png) [@Phil\_Messenger](https://discuss.elastic.co/u/Phil_Messenger)\ **Post date:** [July 17, 2015, 12:21pm UTC](https://discuss.elastic.co/t/filtering-to-a-subset-of-the-full-index/25795/3 "2015-07-17T12:21:22Z")
Will that perform ok if there are say 1,000,000 terms in the filter? Seems like a lot of data to be sending and parsing for each request. I have another issue which is that I don't necessarily have access to a canonical list of ids. Internally we represent them as bloomfilters, so my ideal scenario would be to pass the bloomfilter and use that directly. I found this [https://groups.google.com/forum/#!topic/elasticsearch/dP3R2Gc4J-g](https://groups.google.com/forum/#!topic/elasticsearch/dP3R2Gc4J-g) which got me thinking about a custom filter. Unfortunately Lucene filters are expected to return a DocIdSet which allows doc ids to be iterated. ---
**Author:** ![kohner86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kohner86/32/28260_2.png) [@kohner86](https://discuss.elastic.co/u/kohner86)\ **Post date:** [March 1, 2016, 12:22pm UTC](https://discuss.elastic.co/t/filtering-to-a-subset-of-the-full-index/25795/4 "2016-03-01T12:22:45Z")
A system I'm working on is running into pretty much the exact same issue. How did it turn out? I'm considering doing it via a scripted filter to implement the bloom filter. ---
**Author:** ![Phil\_Messenger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phil_messenger/32/45627_2.png) [@Phil\_Messenger](https://discuss.elastic.co/u/Phil_Messenger)\ **Post date:** [March 2, 2016, 8:01pm UTC](https://discuss.elastic.co/t/filtering-to-a-subset-of-the-full-index/25795/5 "2016-03-02T20:01:33Z")
We got acceptable performance (for our use case) by using a term filter, referencing terms stored in a doc in the index. This meant we avoided the need to sent dozens of megs of JSON around for every query. At the time I couldn't work out how to use a bloomfilter because all of the internal filtering stuff is built around DocSets, and the ability to iterate over them. That's not an operation one can do on a bloomfilter. If I were to look at it again I'd probably look at a compressed bitset - roaringbitmaps for example. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:11pm UTC](https://discuss.elastic.co/t/filtering-to-a-subset-of-the-full-index/25795/6 "2017-07-05T23:11:54Z")