# Threads hanging on BooleanScorer.score with 100% cpu

**URL:** <https://discuss.elastic.co/t/threads-hanging-on-booleanscorer-score-with-100-cpu/14023>\
**Category:** Elasticsearch\
**Created:** [October 18, 2013, 11:57pm UTC](https://discuss.elastic.co/t/threads-hanging-on-booleanscorer-score-with-100-cpu/14023 "2013-10-18T23:57:06Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Josh\_Canfield](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/josh_canfield/32/1705_2.png) [@Josh\_Canfield](https://discuss.elastic.co/u/Josh_Canfield)\
**Post date:** [October 18, 2013, 11:57pm UTC](https://discuss.elastic.co/t/threads-hanging-on-booleanscorer-score-with-100-cpu/14023/1 "2013-10-18T23:57:06Z")

</div>

I'm slowly losing processors as elasticsearch threads hang on  
org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:278)

It happens with "search\_type=dfs\_query\_then\_fetch" and without.  
\*  
\*  
I'm running v0.90.3

Here is a sample query:

{  
"query": {  
"bool": {  
"should": [  
{  
"constant\_score": {  
"query": {  
"text": {  
"brandName": {  
"query": "Paulinie"  
}  
}  
},  
"boost": 3  
}  
},  
{  
"text": {  
"name": {  
"query": "BURNOUT RUFFLES"  
}  
}  
},  
{  
"has\_child": {  
"score\_type": "max",  
"query": {  
"bool": {  
"should": [  
{  
"dis\_max": {  
"queries": [  
{  
"text": {  
"color": {  
"query": "OFFWHITE",  
"boost": 1.5  
}  
}  
},  
{  
"text": {  
"vendors.color": {  
"query": "OFFWHITE",  
"boost": 1.5  
}  
}  
}  
]  
}  
},  
{  
"text": {  
"size": {  
"query": "3T",  
"boost": 1.5  
}  
}  
},  
{  
"text": {  
"sku": {  
"query": "WD2258PA",  
"boost": 5  
}  
}  
},  
{  
"bool": {  
"boost": 10,  
"must": [  
{  
"term": {  
"vendors.vendorId": {  
"value": 5358  
}  
}  
},  
{  
"text": {  
"vendors.sku": {  
"query": "WD2258PA",  
"boost": 2  
}  
}  
}  
]  
}  
}  
]  
}  
},  
"type": "product"  
}  
},  
{  
"fuzzy\_like\_this": {  
"prefix\_length": 2,  
"like\_text": "OFFWHITE",  
"max\_query\_terms": 12,  
"fields": [  
"name"  
]  
}  
}  
]  
}  
},  
"from": 0,  
"size": 10  
}

~ \> curl localhost:9200/\_nodes/hot\_threads

102.0% (510ms out of 500ms) cpu usage by thread  
'elasticsearch[emsweb-02][search][T#16]'  
9/10 snapshots sharing following 16 elements  
org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:278)  
org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:339)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:624)

org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:162)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:488)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:444)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)

org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:134)

org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:295)

org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:175)

org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:144)

org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$2.run(TransportSearchDfsQueryThenFetchAction.java:131)

java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
java.lang.Thread.run(Thread.java:662)  
unique snapshot  
org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:282)  
org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:339)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:624)

org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:162)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:488)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:444)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)

org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:134)

org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:295)

org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:175)

org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:144)

org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$2.run(TransportSearchDfsQueryThenFetchAction.java:131)

java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
java.lang.Thread.run(Thread.java:662)

100.0% (500ms out of 500ms) cpu usage by thread  
'elasticsearch[emsweb-02][search][T#62]'  
10/10 snapshots sharing following 16 elements  
org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:278)  
org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:339)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:624)

org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:162)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:488)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:444)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)  
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)

org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:134)

org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:295)

org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:175)

org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:144)

org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$2.run(TransportSearchDfsQueryThenFetchAction.java:131)

java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
java.lang.Thread.run(Thread.java:662)

Any ideas?

Thanks,  
Josh

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [October 20, 2013, 3:46pm UTC](https://discuss.elastic.co/t/threads-hanging-on-booleanscorer-score-with-100-cpu/14023/2 "2013-10-20T15:46:53Z")

</div>

Hey,

is this 100% reproducible (also with elasticsearch 0.90.5)? If so, can you  
create a bug report on github, which includes the documents you indexed,  
the mapping you created for the index and your queries (including  
dfs\_query\_then\_fetch and other possible settings), so we can reproduce and  
check?

Thanks a lot!

--Alex

On Sat, Oct 19, 2013 at 1:57 AM, Josh Canfield [joshcanfield@gmail.com](mailto:joshcanfield@gmail.com)wrote:

> I'm slowly losing processors as elasticsearch threads hang on  
> org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:278)
> 
> It happens with "search\_type=dfs\_query\_then\_fetch" and without.  
> \*  
> \*  
> I'm running v0.90.3
> 
> Here is a sample query:
> 
> {  
> "query": {  
> "bool": {  
> "should": [  
> {  
> "constant\_score": {  
> "query": {  
> "text": {  
> "brandName": {  
> "query": "Paulinie"  
> }  
> }  
> },  
> "boost": 3  
> }  
> },  
> {  
> "text": {  
> "name": {  
> "query": "BURNOUT RUFFLES"  
> }  
> }  
> },  
> {  
> "has\_child": {  
> "score\_type": "max",  
> "query": {  
> "bool": {  
> "should": [  
> {  
> "dis\_max": {  
> "queries": [  
> {  
> "text": {  
> "color": {  
> "query": "OFFWHITE",  
> "boost": 1.5  
> }  
> }  
> },  
> {  
> "text": {  
> "vendors.color": {  
> "query": "OFFWHITE",  
> "boost": 1.5  
> }  
> }  
> }  
> ]  
> }  
> },  
> {  
> "text": {  
> "size": {  
> "query": "3T",  
> "boost": 1.5  
> }  
> }  
> },  
> {  
> "text": {  
> "sku": {  
> "query": "WD2258PA",  
> "boost": 5  
> }  
> }  
> },  
> {  
> "bool": {  
> "boost": 10,  
> "must": [  
> {  
> "term": {  
> "vendors.vendorId": {  
> "value": 5358  
> }  
> }  
> },  
> {  
> "text": {  
> "vendors.sku": {  
> "query": "WD2258PA",  
> "boost": 2  
> }  
> }  
> }  
> ]  
> }  
> }  
> ]  
> }  
> },  
> "type": "product"  
> }  
> },  
> {  
> "fuzzy\_like\_this": {  
> "prefix\_length": 2,  
> "like\_text": "OFFWHITE",  
> "max\_query\_terms": 12,  
> "fields": [  
> "name"  
> ]  
> }  
> }  
> ]  
> }  
> },  
> "from": 0,  
> "size": 10  
> }
> 
> ~ \> curl localhost:9200/\_nodes/hot\_threads
> 
> 102.0% (510ms out of 500ms) cpu usage by thread  
> 'elasticsearch[emsweb-02][search][T#16]'  
> 9/10 snapshots sharing following 16 elements  
> org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:278)  
> org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:339)
> 
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:624)
> 
> org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:162)
> 
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:488)
> 
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:444)
> 
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
> 
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
> 
> org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:134)
> 
> org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:295)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:175)
> 
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:144)
> 
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$2.run(TransportSearchDfsQueryThenFetchAction.java:131)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> java.lang.Thread.run(Thread.java:662)  
> unique snapshot  
> org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:282)  
> org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:339)
> 
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:624)
> 
> org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:162)
> 
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:488)
> 
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:444)
> 
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
> 
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
> 
> org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:134)
> 
> org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:295)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:175)
> 
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:144)
> 
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$2.run(TransportSearchDfsQueryThenFetchAction.java:131)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> java.lang.Thread.run(Thread.java:662)
> 
> 100.0% (500ms out of 500ms) cpu usage by thread  
> 'elasticsearch[emsweb-02][search][T#62]'  
> 10/10 snapshots sharing following 16 elements  
> org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:278)  
> org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:339)
> 
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:624)
> 
> org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:162)
> 
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:488)
> 
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:444)
> 
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
> 
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
> 
> org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:134)
> 
> org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:295)
> 
> org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:175)
> 
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:144)
> 
> org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$2.run(TransportSearchDfsQueryThenFetchAction.java:131)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> java.lang.Thread.run(Thread.java:662)
> 
> Any ideas?
> 
> Thanks,  
> Josh
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Josh\_Canfield](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/josh_canfield/32/1705_2.png) [@Josh\_Canfield](https://discuss.elastic.co/u/Josh_Canfield)\
**Post date:** [October 20, 2013, 9:06pm UTC](https://discuss.elastic.co/t/threads-hanging-on-booleanscorer-score-with-100-cpu/14023/3 "2013-10-20T21:06:01Z")

</div>

Hi Alex.

I've tried to create a set of documents that would reproduce this. I have  
about 5 million docs, so picking the ones that cause the problem is a bit  
challenging. Worse, the full set of documents aren't causing the problem on  
a test server.

I found a specific query that triggers the 100% CPU usage every time but  
only on my production server. I haven't tried 0.90.5

Here is the query that I could make fail every time, the workaround that I  
finally came up with is changing the highlighted "must" into "should". With  
"must" it hangs, with "should" it worked fine. One variation I tried that  
also hung was if I added "minimum\_should\_match=2" to the modified "should".

The query is dynamically generated based on fields available while  
processing an external file. I'll keep trying to come up with a simple way  
to reproduce.

Here is the query that consistently hung.  
{  
"query": {  
"bool": {  
"should": [  
{  
"constant\_score": {  
"query": {  
"text": {  
"brandName": {  
"query": "R.S.V.P. International"  
}  
}  
},  
"boost": 3  
}  
},  
{  
"text": {  
"name": {  
"query": [  
"Splash Measuring Cup Set"  
],  
"boost": 3  
}  
}  
},  
{  
"has\_child": {  
"score\_type": "max",  
"query": {  
"bool": {  
"should": [  
{  
"bool": {  
"boost": 10,

- 

```
                 **"must"*: [
                  {
                    "text": {
                      "vendors.vendorId": {
                        "query": 7072
                      }
                    }
                  },
                  {
                    "text": {
                      "vendors.sku": {
                        "query": [
                          "ACUP-R"
                        ],
                        "boost": 2
                      }
                    }
                  }
                ]
              }
            }
          ]
        }
      },
      "type": "product"
    }
  }
]

```

}  
},  
"from": 0,  
"size": 1  
}

On Sunday, October 20, 2013 8:46:53 AM UTC-7, Alexander Reelsen wrote:

> Hey,
> 
> is this 100% reproducible (also with elasticsearch 0.90.5)? If so, can you  
> create a bug report on github, which includes the documents you indexed,  
> the mapping you created for the index and your queries (including  
> dfs\_query\_then\_fetch and other possible settings), so we can reproduce and  
> check?
> 
> Thanks a lot!
> 
> --Alex
> 
> On Sat, Oct 19, 2013 at 1:57 AM, Josh Canfield \<[joshca...@gmail.com](mailto:joshca...@gmail.com)\<javascript:\>
> 
> > wrote:
> 
> > I'm slowly losing processors as elasticsearch threads hang on  
> > org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:278)
> > 
> > It happens with "search\_type=dfs\_query\_then\_fetch" and without.  
> > \*  
> > \*  
> > I'm running v0.90.3
> > 
> > Here is a sample query:
> > 
> > {  
> > "query": {  
> > "bool": {  
> > "should": [  
> > {  
> > "constant\_score": {  
> > "query": {  
> > "text": {  
> > "brandName": {  
> > "query": "Paulinie"  
> > }  
> > }  
> > },  
> > "boost": 3  
> > }  
> > },  
> > {  
> > "text": {  
> > "name": {  
> > "query": "BURNOUT RUFFLES"  
> > }  
> > }  
> > },  
> > {  
> > "has\_child": {  
> > "score\_type": "max",  
> > "query": {  
> > "bool": {  
> > "should": [  
> > {  
> > "dis\_max": {  
> > "queries": [  
> > {  
> > "text": {  
> > "color": {  
> > "query": "OFFWHITE",  
> > "boost": 1.5  
> > }  
> > }  
> > },  
> > {  
> > "text": {  
> > "vendors.color": {  
> > "query": "OFFWHITE",  
> > "boost": 1.5  
> > }  
> > }  
> > }  
> > ]  
> > }  
> > },  
> > {  
> > "text": {  
> > "size": {  
> > "query": "3T",  
> > "boost": 1.5  
> > }  
> > }  
> > },  
> > {  
> > "text": {  
> > "sku": {  
> > "query": "WD2258PA",  
> > "boost": 5  
> > }  
> > }  
> > },  
> > {  
> > "bool": {  
> > "boost": 10,  
> > "must": [  
> > {  
> > "term": {  
> > "vendors.vendorId": {  
> > "value": 5358  
> > }  
> > }  
> > },  
> > {  
> > "text": {  
> > "vendors.sku": {  
> > "query": "WD2258PA",  
> > "boost": 2  
> > }  
> > }  
> > }  
> > ]  
> > }  
> > }  
> > ]  
> > }  
> > },  
> > "type": "product"  
> > }  
> > },  
> > {  
> > "fuzzy\_like\_this": {  
> > "prefix\_length": 2,  
> > "like\_text": "OFFWHITE",  
> > "max\_query\_terms": 12,  
> > "fields": [  
> > "name"  
> > ]  
> > }  
> > }  
> > ]  
> > }  
> > },  
> > "from": 0,  
> > "size": 10  
> > }
> > 
> > ~ \> curl localhost:9200/\_nodes/hot\_threads
> > 
> > 102.0% (510ms out of 500ms) cpu usage by thread  
> > 'elasticsearch[emsweb-02][search][T#16]'  
> > 9/10 snapshots sharing following 16 elements
> > 
> > org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:278)
> > 
> > org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:339)
> > 
> > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:624)
> > 
> > org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:162)
> > 
> > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:488)
> > 
> > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:444)
> > 
> > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
> > 
> > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
> > 
> > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:134)
> > 
> > org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:295)
> > 
> > org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:175)
> > 
> > org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:144)
> > 
> > org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$2.run(TransportSearchDfsQueryThenFetchAction.java:131)
> > 
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > 
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > java.lang.Thread.run(Thread.java:662)  
> > unique snapshot
> > 
> > org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:282)
> > 
> > org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:339)
> > 
> > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:624)
> > 
> > org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:162)
> > 
> > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:488)
> > 
> > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:444)
> > 
> > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
> > 
> > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
> > 
> > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:134)
> > 
> > org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:295)
> > 
> > org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:175)
> > 
> > org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:144)
> > 
> > org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$2.run(TransportSearchDfsQueryThenFetchAction.java:131)
> > 
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > 
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > java.lang.Thread.run(Thread.java:662)
> > 
> > 100.0% (500ms out of 500ms) cpu usage by thread  
> > 'elasticsearch[emsweb-02][search][T#62]'  
> > 10/10 snapshots sharing following 16 elements
> > 
> > org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:278)
> > 
> > org.apache.lucene.search.BooleanScorer.score(BooleanScorer.java:339)
> > 
> > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:624)
> > 
> > org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:162)
> > 
> > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:488)
> > 
> > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:444)
> > 
> > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
> > 
> > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
> > 
> > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:134)
> > 
> > org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:295)
> > 
> > org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:175)
> > 
> > org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction.executeQuery(TransportSearchDfsQueryThenFetchAction.java:144)
> > 
> > org.elasticsearch.action.search.type.TransportSearchDfsQueryThenFetchAction$AsyncAction$2.run(TransportSearchDfsQueryThenFetchAction.java:131)
> > 
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > 
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > java.lang.Thread.run(Thread.java:662)
> > 
> > Any ideas?
> > 
> > Thanks,  
> > Josh
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\
**Post date:** [October 21, 2013, 8:08am UTC](https://discuss.elastic.co/t/threads-hanging-on-booleanscorer-score-with-100-cpu/14023/4 "2013-10-21T08:08:18Z")

</div>

Josh,

Indeed, this can be tricky to reproduce since this kind of issues may  
depend on how your index is splitted into segments and on the order of  
documents in individual segments. Maybe you can try to run this query with  
assertions enabled on the JVM to see if this triggers something (running  
`export JAVA_OPTS="-ea"` before starting elasticsearch from the same  
terminal would work)? Otherwise, would it be doable to share the faulty  
shard?

--  
Adrien Grand

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Josh\_Canfield](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/josh_canfield/32/1705_2.png) [@Josh\_Canfield](https://discuss.elastic.co/u/Josh_Canfield)\
**Post date:** [October 21, 2013, 10:45pm UTC](https://discuss.elastic.co/t/threads-hanging-on-booleanscorer-score-with-100-cpu/14023/5 "2013-10-21T22:45:38Z")

</div>

Hi Adrien and group.

I tried enabling asserts but none are getting triggered. What I've found is  
that somehow the BooleanScorer's bucketTable data structure is getting  
corrupted. I'm getting a loop in the linked list so it's never leaving the  
score method. I've been able to reproduce this consistently on my test  
server now but I don't have a dataset that I can share yet.

This may be a Lucene issue.

On Mon, Oct 21, 2013 at 1:08 AM, Adrien Grand \<  
[adrien.grand@elasticsearch.com](mailto:adrien.grand@elasticsearch.com)\> wrote:

> Josh,
> 
> Indeed, this can be tricky to reproduce since this kind of issues may  
> depend on how your index is splitted into segments and on the order of  
> documents in individual segments. Maybe you can try to run this query with  
> assertions enabled on the JVM to see if this triggers something (running  
> `export JAVA_OPTS="-ea"` before starting elasticsearch from the same  
> terminal would work)? Otherwise, would it be doable to share the faulty  
> shard?
> 
> --  
> Adrien Grand
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\
**Post date:** [October 23, 2013, 4:09pm UTC](https://discuss.elastic.co/t/threads-hanging-on-booleanscorer-score-with-100-cpu/14023/6 "2013-10-23T16:09:30Z")

</div>

On Tue, Oct 22, 2013 at 12:45 AM, Josh Canfield [joshcanfield@gmail.com](mailto:joshcanfield@gmail.com)wrote:

> Hi Adrien and group.
> 
> I tried enabling asserts but none are getting triggered. What I've found  
> is that somehow the BooleanScorer's bucketTable data structure is getting  
> corrupted. I'm getting a loop in the linked list so it's never leaving the  
> score method. I've been able to reproduce this consistently on my test  
> server now but I don't have a dataset that I can share yet.
> 
> This may be a Lucene issue.

Indeed, there is a bug somewhere. My guess would be either the boolean  
scorer or maybe one of its child scorers breaks a contract on some API  
(maybe more likely?). One last option would be that it is a JVM bug, in  
which case it'd be useful to check that you can reproduce with Java 1.7  
u25, which has no known bug related to Lucene I think (or at least it has  
fewer bugs than other JVMs).

--  
Adrien Grand

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Josh\_Canfield](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/josh_canfield/32/1705_2.png) [@Josh\_Canfield](https://discuss.elastic.co/u/Josh_Canfield)\
**Post date:** [October 23, 2013, 9:58pm UTC](https://discuss.elastic.co/t/threads-hanging-on-booleanscorer-score-with-100-cpu/14023/7 "2013-10-23T21:58:45Z")

</div>

Ok, I've found and fixed the defect and created a ticket.

> <https://github.com/elastic/elasticsearch/issues/3955>
>
> The defect is in 
> ChildrenQuery.ParentScorer#nextDoc()
> 
> \`\`\` java
> @Over…ride
> public int nextDoc() throws IOException {
> if (remaining == 0) {
> return NO\_MORE\_DOCS;
> }
> ...
> \`\`\`
> 
> This fails to update \`currentDocId\`. This causes \`ChildrenQuery.ParentScorer#docID()\` to continue returning the last doc id forever.
> 
> The BooleanScorer calls
> 
> \`\`\` java
> public boolean score(Collector collector, int max, int firstDocID)
> \`\`\`
> 
> passing the \`firstDocID\` it pulled from the \`docID()\`. When the ParentScorer finishes with a low id document but continues to get processed and added to the BucketTable it opens the possibility of a document from the other boolean sub queries overlapping and causing get an infinite loop.
> 
> The fix is to set 
> \`currentDocId = NO\_MORE\_DOCS;\`
> in both \`int nextDoc()\` and \`int advance(int target)\`

Josh

On Wed, Oct 23, 2013 at 9:09 AM, Adrien Grand \<  
[adrien.grand@elasticsearch.com](mailto:adrien.grand@elasticsearch.com)\> wrote:

> On Tue, Oct 22, 2013 at 12:45 AM, Josh Canfield [joshcanfield@gmail.com](mailto:joshcanfield@gmail.com)wrote:
> 
> > Hi Adrien and group.
> > 
> > I tried enabling asserts but none are getting triggered. What I've found  
> > is that somehow the BooleanScorer's bucketTable data structure is getting  
> > corrupted. I'm getting a loop in the linked list so it's never leaving the  
> > score method. I've been able to reproduce this consistently on my test  
> > server now but I don't have a dataset that I can share yet.
> > 
> > This may be a Lucene issue.
> 
> Indeed, there is a bug somewhere. My guess would be either the boolean  
> scorer or maybe one of its child scorers breaks a contract on some API  
> (maybe more likely?). One last option would be that it is a JVM bug, in  
> which case it'd be useful to check that you can reproduce with Java 1.7  
> u25, which has no known bug related to Lucene I think (or at least it has  
> fewer bugs than other JVMs).
> 
> --  
> Adrien Grand
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\
**Post date:** [October 23, 2013, 10:38pm UTC](https://discuss.elastic.co/t/threads-hanging-on-booleanscorer-score-with-100-cpu/14023/8 "2013-10-23T22:38:59Z")

</div>

On Wed, Oct 23, 2013 at 11:58 PM, Josh Canfield [joshcanfield@gmail.com](mailto:joshcanfield@gmail.com)wrote:

> Ok, I've found and fixed the defect and created a ticket.
> 
> [has\_child can cause an infinite loop (100% CPU) when used in bool query · Issue #3955 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/3955)

Wow, I'm so happy that you managed to track it down! The fix looks good, I  
just assigned the issue to myself and will take some more time tomorrow to  
test your fix and see if we can prevent this kind of error from happening  
again in the future.

Thank you again!

--  
Adrien Grand

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Josh\_Canfield](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/josh_canfield/32/1705_2.png) [@Josh\_Canfield](https://discuss.elastic.co/u/Josh_Canfield)\
**Post date:** [October 23, 2013, 11:28pm UTC](https://discuss.elastic.co/t/threads-hanging-on-booleanscorer-score-with-100-cpu/14023/9 "2013-10-23T23:28:34Z")

</div>

> Wow, I'm so happy that you managed to track it down! The fix looks good, I  
> just assigned the issue to myself and will take some more time tomorrow to  
> test your fix and see if we can prevent this kind of error from happening  
> again in the future.

Thank you again!

Me too! My users were getting frustrated with the poor results that  
no-scoring was giving 🙂

On Wed, Oct 23, 2013 at 3:38 PM, Adrien Grand \<  
[adrien.grand@elasticsearch.com](mailto:adrien.grand@elasticsearch.com)\> wrote:

> On Wed, Oct 23, 2013 at 11:58 PM, Josh Canfield [joshcanfield@gmail.com](mailto:joshcanfield@gmail.com)wrote:
> 
> > Ok, I've found and fixed the defect and created a ticket.
> > 
> > [has\_child can cause an infinite loop (100% CPU) when used in bool query · Issue #3955 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/3955)
> 
> Wow, I'm so happy that you managed to track it down! The fix looks good, I  
> just assigned the issue to myself and will take some more time tomorrow to  
> test your fix and see if we can prevent this kind of error from happening  
> again in the future.
> 
> Thank you again!
> 
> --  
> Adrien Grand
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:10am UTC](https://discuss.elastic.co/t/threads-hanging-on-booleanscorer-score-with-100-cpu/14023/10 "2017-07-06T02:10:58Z")

</div>