# Poor performance of scripted bitwise operations in facet filters **URL:** \ **Category:** Elasticsearch\ **Created:** [April 12, 2012, 2:28pm UTC](https://discuss.elastic.co/t/poor-performance-of-scripted-bitwise-operations-in-facet-filters/7308 "2012-04-12T14:28:37Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Shane\_Witbeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_witbeck/32/2803_2.png) [@Shane\_Witbeck](https://discuss.elastic.co/u/Shane_Witbeck)\ **Post date:** [April 12, 2012, 2:28pm UTC](https://discuss.elastic.co/t/poor-performance-of-scripted-bitwise-operations-in-facet-filters/7308/1 "2012-04-12T14:28:37Z")
I have a requirement where several facets need to do bitwise operations similar to the following: "opened\_today\_dev":{ "query":{ "query\_string":{ "query":"categoryID:1 AND dateCreated:[2012-04-12 TO \*]" } }, "facet\_filter" : { "script":{ "script":"(doc['owner.userAccessTypeBit'].intValue & 8) \> 0" } } } I've found that the face\_filter adds a huge performance hit to the overall query which has several other facets. This is unacceptable since I'm running the query every five seconds or so. 1. Is there a more efficient way of doing bitwise operations? 2. I'm also curious if splitting up the bitmask (owner.userAccessTypeBit) above into individual bits and using a terms filter would be better? Thanks, Shane ---
**Author:** ![Dan\_Everton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_everton/32/1865_2.png) [@Dan\_Everton](https://discuss.elastic.co/u/Dan_Everton)\ **Post date:** [April 12, 2012, 11:27pm UTC](https://discuss.elastic.co/t/poor-performance-of-scripted-bitwise-operations-in-facet-filters/7308/2 "2012-04-12T23:27:55Z")
Script facets are going to be slow no matter what you do. You could make it more efficient, but it's still going to execute that script for each matching document. Splitting the bitmask and using a term filter will be much, much faster as you're using what ES is good it, search, rather than script execution. Cheers, Dan ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 13, 2012, 12:41pm UTC](https://discuss.elastic.co/t/poor-performance-of-scripted-bitwise-operations-in-facet-filters/7308/3 "2012-04-13T12:41:29Z")
Dan point is good. I would add that you need to make sure you are not counting the cost of loading the field to memory when you do the perf tests. Also, it would be interesting to know how many total\_hits that filter executed on? Effectively, this is a CPU intensive execution, and if its slow, one can always write a native Java implementation of that script, though a terms/term filter will probably be better since it can be effectively cached. On Fri, Apr 13, 2012 at 2:27 AM, Dan Everton [dan@iocaine.org](mailto:dan@iocaine.org) wrote: > Script facets are going to be slow no matter what you do. You could make > it more efficient, but it's still going to execute that script for each > matching document. > > Splitting the bitmask and using a term filter will be much, much faster as > you're using what ES is good it, search, rather than script execution. > > Cheers, > Dan ---
**Author:** ![Shane\_Witbeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_witbeck/32/2803_2.png) [@Shane\_Witbeck](https://discuss.elastic.co/u/Shane_Witbeck)\ **Post date:** [April 13, 2012, 1:02pm UTC](https://discuss.elastic.co/t/poor-performance-of-scripted-bitwise-operations-in-facet-filters/7308/4 "2012-04-13T13:02:23Z")
Thanks again. I decided to go with splitting the bitmask and using a terms filter. Performance is much better although doing a bitwise operation in a querystring query would have been ideal. Although, I believe this is more a Lucene limitation. On Friday, April 13, 2012 8:41:29 AM UTC-4, kimchy wrote: > Dan point is good. I would add that you need to make sure you are not > counting the cost of loading the field to memory when you do the perf > tests. Also, it would be interesting to know how many total\_hits that > filter executed on? Effectively, this is a CPU intensive execution, and if > its slow, one can always write a native Java implementation of that script, > though a terms/term filter will probably be better since it can be > effectively cached. > > On Fri, Apr 13, 2012 at 2:27 AM, Dan Everton [dan@iocaine.org](mailto:dan@iocaine.org) wrote: > > > Script facets are going to be slow no matter what you do. You could make > > it more efficient, but it's still going to execute that script for each > > matching document. > > > > Splitting the bitmask and using a term filter will be much, much faster > > as you're using what ES is good it, search, rather than script execution. > > > > Cheers, > > Dan ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 14, 2012, 5:27pm UTC](https://discuss.elastic.co/t/poor-performance-of-scripted-bitwise-operations-in-facet-filters/7308/5 "2012-04-14T17:27:01Z")
Great. Just a note, its not really a Lucene limitation, scripting is implemented on the elasticsearch level, and it will be CPU bound as it needs to compute the script for each matching document. On Fri, Apr 13, 2012 at 4:02 PM, Shane Witbeck [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)wrote: > Thanks again. I decided to go with splitting the bitmask and using a terms > filter. Performance is much better although doing a bitwise operation in a > querystring query would have been ideal. Although, I believe this is more a > Lucene limitation. > > On Friday, April 13, 2012 8:41:29 AM UTC-4, kimchy wrote: > > > Dan point is good. I would add that you need to make sure you are not > > counting the cost of loading the field to memory when you do the perf > > tests. Also, it would be interesting to know how many total\_hits that > > filter executed on? Effectively, this is a CPU intensive execution, and if > > its slow, one can always write a native Java implementation of that script, > > though a terms/term filter will probably be better since it can be > > effectively cached. > > > > On Fri, Apr 13, 2012 at 2:27 AM, Dan Everton [dan@iocaine.org](mailto:dan@iocaine.org) wrote: > > > > > Script facets are going to be slow no matter what you do. You could make > > > it more efficient, but it's still going to execute that script for each > > > matching document. > > > > > > Splitting the bitmask and using a term filter will be much, much faster > > > as you're using what ES is good it, search, rather than script execution. > > > > > > Cheers, > > > Dan ---
**Author:** ![Shane\_Witbeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_witbeck/32/2803_2.png) [@Shane\_Witbeck](https://discuss.elastic.co/u/Shane_Witbeck)\ **Post date:** [April 16, 2012, 3:09pm UTC](https://discuss.elastic.co/t/poor-performance-of-scripted-bitwise-operations-in-facet-filters/7308/6 "2012-04-16T15:09:10Z")
I wasn't clear on what I meant about the Lucene limitation. What I meant was support for bitwise operations. Is this something I should add as a feature request for ES or would it be better suited for a Lucene feature request? On Saturday, April 14, 2012 1:27:01 PM UTC-4, kimchy wrote: > Great. Just a note, its not really a Lucene limitation, scripting is > implemented on the elasticsearch level, and it will be CPU bound as it > needs to compute the script for each matching document. > > On Fri, Apr 13, 2012 at 4:02 PM, Shane Witbeck [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)wrote: > > > Thanks again. I decided to go with splitting the bitmask and using a > > terms filter. Performance is much better although doing a bitwise operation > > in a querystring query would have been ideal. Although, I believe this is > > more a Lucene limitation. > > > > On Friday, April 13, 2012 8:41:29 AM UTC-4, kimchy wrote: > > > > > Dan point is good. I would add that you need to make sure you are not > > > counting the cost of loading the field to memory when you do the perf > > > tests. Also, it would be interesting to know how many total\_hits that > > > filter executed on? Effectively, this is a CPU intensive execution, and if > > > its slow, one can always write a native Java implementation of that script, > > > though a terms/term filter will probably be better since it can be > > > effectively cached. > > > > > > On Fri, Apr 13, 2012 at 2:27 AM, Dan Everton [dan@iocaine.org](mailto:dan@iocaine.org) wrote: > > > > > > > Script facets are going to be slow no matter what you do. You could > > > > make it more efficient, but it's still going to execute that script for > > > > each matching document. > > > > > > > > Splitting the bitmask and using a term filter will be much, much faster > > > > as you're using what ES is good it, search, rather than script execution. > > > > > > > > Cheers, > > > > Dan ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 17, 2012, 1:13pm UTC](https://discuss.elastic.co/t/poor-performance-of-scripted-bitwise-operations-in-facet-filters/7308/7 "2012-04-17T13:13:42Z")
bitwise can be implemented as a built in filter, but effectively it will do something similar to what you did in the script, just in Java (which you can do yourself as well by writing a native Java script module). On Mon, Apr 16, 2012 at 6:09 PM, Shane Witbeck [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)wrote: > I wasn't clear on what I meant about the Lucene limitation. What I meant > was support for bitwise operations. > > Is this something I should add as a feature request for ES or would it be > better suited for a Lucene feature request? > > On Saturday, April 14, 2012 1:27:01 PM UTC-4, kimchy wrote: > > > Great. Just a note, its not really a Lucene limitation, scripting is > > implemented on the elasticsearch level, and it will be CPU bound as it > > needs to compute the script for each matching document. > > > > On Fri, Apr 13, 2012 at 4:02 PM, Shane Witbeck [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)wrote: > > > > > Thanks again. I decided to go with splitting the bitmask and using a > > > terms filter. Performance is much better although doing a bitwise operation > > > in a querystring query would have been ideal. Although, I believe this is > > > more a Lucene limitation. > > > > > > On Friday, April 13, 2012 8:41:29 AM UTC-4, kimchy wrote: > > > > > > > Dan point is good. I would add that you need to make sure you are not > > > > counting the cost of loading the field to memory when you do the perf > > > > tests. Also, it would be interesting to know how many total\_hits that > > > > filter executed on? Effectively, this is a CPU intensive execution, and if > > > > its slow, one can always write a native Java implementation of that script, > > > > though a terms/term filter will probably be better since it can be > > > > effectively cached. > > > > > > > > On Fri, Apr 13, 2012 at 2:27 AM, Dan Everton [dan@iocaine.org](mailto:dan@iocaine.org) wrote: > > > > > > > > > Script facets are going to be slow no matter what you do. You could > > > > > make it more efficient, but it's still going to execute that script for > > > > > each matching document. > > > > > > > > > > Splitting the bitmask and using a term filter will be much, much > > > > > faster as you're using what ES is good it, search, rather than script > > > > > execution. > > > > > > > > > > Cheers, > > > > > Dan ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:32am UTC](https://discuss.elastic.co/t/poor-performance-of-scripted-bitwise-operations-in-facet-filters/7308/8 "2017-07-06T03:32:17Z")