# Bloom filter codec? **URL:** \ **Category:** Elasticsearch\ **Created:** [July 17, 2014, 2:25pm UTC](https://discuss.elastic.co/t/bloom-filter-codec/18717 "2014-07-17T14:25:11Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [July 17, 2014, 2:25pm UTC](https://discuss.elastic.co/t/bloom-filter-codec/18717/1 "2014-07-17T14:25:11Z")
Has anyone had success adding a bloom filter to the codec for any of their fields? [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-codec.html#bloom-postings](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-codec.html#bloom-postings) I imagine it'd help reduce IO from (non multi-term) queries that frequently don't match. Like if you have a field that is very specific and useful for searching but very rarely matches anything. It looks like the cost is in the range of 10 bits of heap per term per segment for a false positive probability around 1%. Meaning it'd be pretty high if the index had lots of terms - especially if they were in many segments. But it'd be about 10 bits per value if the values were mostly unique. Nik -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3X11bwogWi9oFTYFzzO6%2BdnvsOqcEFWG\_dB5c%2Boy%3D4Fw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3X11bwogWi9oFTYFzzO6%2BdnvsOqcEFWG_dB5c%2Boy%3D4Fw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [July 17, 2014, 8:31pm UTC](https://discuss.elastic.co/t/bloom-filter-codec/18717/2 "2014-07-17T20:31:40Z")
Hi Nik, The trade-off is not easy indeed. First, the default terms dictionary can already save some disk seeks. By storing the prefixes of the terms that are in the terms dictionary in a FST in memory, it can avoid going to disk when the term that you are looking up cannot match this FST. A bloom filter might save a few additional disk seeks but as you said, it's pretty intensive memory-wise and sometimes that is memory that would just be better spent on the filesystem cache. On Thu, Jul 17, 2014 at 4:25 PM, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com) wrote: > Has anyone had success adding a bloom filter to the codec for any of their > fields? > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-codec.html#bloom-postings) > > I imagine it'd help reduce IO from (non multi-term) queries that > frequently don't match. Like if you have a field that is very specific and > useful for searching but very rarely matches anything. > > It looks like the cost is in the range of 10 bits of heap per term per > segment for a false positive probability around 1%. Meaning it'd be pretty > high if the index had lots of terms - especially if they were in many > segments. But it'd be about 10 bits per value if the values were mostly > unique. > > Nik > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3X11bwogWi9oFTYFzzO6%2BdnvsOqcEFWG\_dB5c%2Boy%3D4Fw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3X11bwogWi9oFTYFzzO6%2BdnvsOqcEFWG_dB5c%2Boy%3D4Fw%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3X11bwogWi9oFTYFzzO6%2BdnvsOqcEFWG\_dB5c%2Boy%3D4Fw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3X11bwogWi9oFTYFzzO6%2BdnvsOqcEFWG_dB5c%2Boy%3D4Fw%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j52TNTaN8NzNpB5jd-Kms3VuVtn\_0ZFVqbt%2B7tfhk%3D1WQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j52TNTaN8NzNpB5jd-Kms3VuVtn_0ZFVqbt%2B7tfhk%3D1WQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [July 17, 2014, 8:37pm UTC](https://discuss.elastic.co/t/bloom-filter-codec/18717/3 "2014-07-17T20:37:14Z")
Thanks for replying. I've been looking to reduce my IO. Pushing everything into an all field is really going to be the biggest thing, I think, but I was wondering about the bloom filters. It doesn't sound worth it. It feels like everything but the default codec is pretty unlikely to be useful? On Thu, Jul 17, 2014 at 4:31 PM, Adrien Grand \< [adrien.grand@elasticsearch.com](mailto:adrien.grand@elasticsearch.com)\> wrote: > Hi Nik, > > The trade-off is not easy indeed. First, the default terms dictionary can > already save some disk seeks. By storing the prefixes of the terms that are > in the terms dictionary in a FST in memory, it can avoid going to disk when > the term that you are looking up cannot match this FST. A bloom filter > might save a few additional disk seeks but as you said, it's pretty > intensive memory-wise and sometimes that is memory that would just be > better spent on the filesystem cache. > > On Thu, Jul 17, 2014 at 4:25 PM, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com) > wrote: > > > Has anyone had success adding a bloom filter to the codec for any of > > their fields? > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-codec.html#bloom-postings) > > > > I imagine it'd help reduce IO from (non multi-term) queries that > > frequently don't match. Like if you have a field that is very specific and > > useful for searching but very rarely matches anything. > > > > It looks like the cost is in the range of 10 bits of heap per term per > > segment for a false positive probability around 1%. Meaning it'd be pretty > > high if the index had lots of terms - especially if they were in many > > segments. But it'd be about 10 bits per value if the values were mostly > > unique. > > > > Nik > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3X11bwogWi9oFTYFzzO6%2BdnvsOqcEFWG\_dB5c%2Boy%3D4Fw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3X11bwogWi9oFTYFzzO6%2BdnvsOqcEFWG_dB5c%2Boy%3D4Fw%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3X11bwogWi9oFTYFzzO6%2BdnvsOqcEFWG\_dB5c%2Boy%3D4Fw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3X11bwogWi9oFTYFzzO6%2BdnvsOqcEFWG_dB5c%2Boy%3D4Fw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Adrien Grand > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j52TNTaN8NzNpB5jd-Kms3VuVtn\_0ZFVqbt%2B7tfhk%3D1WQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j52TNTaN8NzNpB5jd-Kms3VuVtn_0ZFVqbt%2B7tfhk%3D1WQ%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j52TNTaN8NzNpB5jd-Kms3VuVtn\_0ZFVqbt%2B7tfhk%3D1WQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j52TNTaN8NzNpB5jd-Kms3VuVtn_0ZFVqbt%2B7tfhk%3D1WQ%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0dpqMAkLZ%3DOdWfhicO9hcB5ummBrnmTPw7xUG-54G1pQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0dpqMAkLZ%3DOdWfhicO9hcB5ummBrnmTPw7xUG-54G1pQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [July 17, 2014, 10:49pm UTC](https://discuss.elastic.co/t/bloom-filter-codec/18717/4 "2014-07-17T22:49:29Z")
On Thu, Jul 17, 2014 at 10:37 PM, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com) wrote: > Thanks for replying. I've been looking to reduce my IO. Pushing > everything into an all field is really going to be the biggest thing, I > think, but I was wondering about the bloom filters. It doesn't sound worth > it. It feels like everything but the default codec is pretty unlikely to > be useful? Indeed, the default codec tries to make sensible trade-offs and would be the most useful in most cases. -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7ou5PLbCJ744nP\_Qk\_S5mwfrbrPUxG0dGkn9zYiQwrsw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j7ou5PLbCJ744nP_Qk_S5mwfrbrPUxG0dGkn9zYiQwrsw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Arvind\_Kumar\_Chigura](https://avatars.discourse-cdn.com/v4/letter/a/838e76/32.png) [@Arvind\_Kumar\_Chigura](https://discuss.elastic.co/u/Arvind_Kumar_Chigura)\ **Post date:** [May 13, 2016, 3:17pm UTC](https://discuss.elastic.co/t/bloom-filter-codec/18717/5 "2016-05-13T15:17:06Z")
> [@jpountz](#): > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-codec.html#bloom-postings) Are bloom filters still supported in 2.3 ? ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [May 13, 2016, 3:28pm UTC](https://discuss.elastic.co/t/bloom-filter-codec/18717/6 "2016-05-13T15:28:02Z")
No, they are not. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:51pm UTC](https://discuss.elastic.co/t/bloom-filter-codec/18717/7 "2017-07-05T22:51:45Z")