Null_pointer_exception when use bool query

Hi everyone. I have the weird issue when i use simple bool query

curl -X GET "localhost:9200/people/_search" -H 'Content-Type: application/json' -d'
{
			"query": {
				"bool": {
					"should": [
						{"match":{"name":"andrew"}}
					],
					"must": [
					    {"match":{"birth_year":"2005"}}
					]
				}
			},

	                "sort":{"_score":{"order":"desc"}},"from":0,"size":"15","min_score":5
}
' | python -m json.tool

In this case i get null_pointer_exception error. The weird thing is, with different options i may not get it, so, for example with birth_year = 2019 i don't get it. I thought that this can be because of wrong usage of bool query or wrong settings or something like that, but wasn't managed to find why exactly that happens. I fixed that issue using "filter" clause instead of "must" - this way that works and i don't get the error, but, that is not a solution.

ES version which i use: 7.1.0

docker-compose.yml:

elasticsearch:
        image: elasticsearch:7.1.0
        restart: always
        ports:
            - "9200:9200"
            - "9300:9300"
        volumes:
            - ./esdata71:/usr/share/elasticsearch/data
        environment:
            - bootstrap.memory_lock=true
            - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
            - discovery.type=single-node
        ulimits:
            memlock:
                soft: -1
                hard: -1

Index settings:

[
        "max_ngram_diff" => 10,
        "analysis" => [
            "tokenizer" => [
                "edge_trigrams_tokenizer" => [
                    "type"    => "edge_ngram",
                    "min_gram"=> 3,
                    "max_gram"=> 20,
                    "token_chars" => ["letter", "whitespace", "punctuation", "symbol", "digit"]

                ],
                "trigrams_tokenizer" => [
                    "type"    => "ngram",
                    "min_gram"=> 3,
                    "max_gram"=> 10,
                    "token_chars" => ["letter", "whitespace", "punctuation", "symbol", "digit"]
                ],
                "short_trigrams_tokenizer" => [
                    "type" => "edge_ngram",
                    "min_gram" => 1,
                    "max_gram" => 3,
                    "token_chars" => ["letter", "whitespace", "punctuation", "symbol", "digit"]
                ]
            ],
            "analyzer"=> [
                "edge_trigrams"=> [
                    "type"=>  "custom",
                    "tokenizer"=> "edge_trigrams_tokenizer",
                    "filter"=>   [
                        "lowercase", "asciifolding"
                    ],
                    "char_filter" => ["synonym"]
                ],
                "trigrams"=> [
                    "type"=>  "custom",
                    "tokenizer"=> "trigrams_tokenizer",
                    "filter"=>   [
                        "lowercase", "asciifolding"
                    ],
                    "char_filter" => ["synonym"]
                ]
            ],
            "char_filter" => [
                "synonym" => [
                    "type" => "mapping",
                    "mappings" => [
                        "v.d => van de",
                        "v/d => van de",
                        "vd => van de",
                        "vh => van het",
                        "v/h => van het",
                        "v.h => van het",
                        "dl => de la",
                        "d/l => de la",
                        "d'la => de la",
                        "du => de le",
                        "de l' => de la"
                    ]
                ]
            ]
        ]
    ]

Mapping settings:

    "name"        => [
        'type'      => 'text',
        'analyzer'  => 'edge_trigrams'
    ],
    "last_name" => [
        'type'      => 'text',
        'analyzer'  => 'edge_trigrams'
    ],
    "exact_name" => [
        'type'      => 'text',
        'analyzer'  => 'trigrams'
    ],
    "birth_year" => [
        'type'      => 'text',
        'analyzer'  => 'standard'
    ],
    "sex" => [
        'type'      => 'text',
        'analyzer'  => 'standard'
    ],
    "active" => [
        'type'      => 'boolean'
    ]

The error which i get:

{
    "error": {
        "caused_by": {
            "caused_by": {
                "reason": null,
                "type": "null_pointer_exception"
            },
            "reason": null,
            "type": "null_pointer_exception"
        },
        "failed_shards": [
            {
                "index": "people",
                "node": "3beKgQxDRxe5V5Ln6ZwmUA",
                "reason": {
                    "reason": null,
                    "type": "null_pointer_exception"
                },
                "shard": 0
            }
        ],
        "grouped": true,
        "phase": "query",
        "reason": "all shards failed",
        "root_cause": [
            {
                "reason": null,
                "type": "null_pointer_exception"
            }
        ],
        "type": "search_phase_execution_exception"
    },
    "status": 500
}

Another weird thing, even with the same options (like the same name and birth_year) but with different "min_score" in query, in one case i get error but with different "min_score" i don't get error.

I assume that this is something related to score calculation, or to some kind of data manipulation inside the ES engine, so probably something wrong inside my index and that cause an exception in different specific cases, so basically some documents inside my index have something which should not be there.

UPDATE:
After investigating i figured out next:
That happens when i use "min_score" in my query

Data from Elasticsearch log:

"Caused by: java.lang.NullPointerException",
"at org.apache.lucene.search.ReqOptSumScorer.setMinCompetitiveScore(ReqOptSumScorer.java:299) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]",
"at org.apache.lucene.search.ScoreCachingWrappingScorer.setMinCompetitiveScore(ScoreCachingWrappingScorer.java:59) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]",
"at org.apache.lucene.search.TopScoreDocCollector.updateMinCompetitiveScore(TopScoreDocCollector.java:242) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]",
"at org.apache.lucene.search.TopScoreDocCollector$SimpleTopScoreDocCollector$1.collect(TopScoreDocCollector.java:78) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]",
"at org.elasticsearch.common.lucene.MinimumScoreCollector.collect(MinimumScoreCollector.java:57) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:263) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]",
"at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:214) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]",
"at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]",
"at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:652) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]",
"at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:177) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:443) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]",
"at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:275) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:115) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:349) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:393) ~[elasticsearch-7.1.0.jar:7.1.0]",
"at org.elasticsearch.search.SearchService.access$100(SearchService.java:124) ~[elasticsearch-7.1.0.jar:7.1.0]",

So, from the top line i noticed that error appears in the file ReqOptSumScorer.java:299

Here is repository:

and on the 299 line i see next:
maxScorePropagator.setMinCompetitiveScore(minScore);

variable is defined on line 54:

if (scoreMode == ScoreMode.TOP_SCORES) {
     this.maxScorePropagator = new MaxScoreSumPropagator(Arrays.asList(reqScorer, optScorer));
} else {
     this.maxScorePropagator = null;
}

so i assume that in my case it goes to "else" section and variable is NULL.

Then i found info about "score_mode" in elasticsearch documentation:
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html
So, there are several possible variants of "score_mode", but when i change query to this:

curl -X GET "localhost:9200/people/_search" -H 'Content-Type: application/json' -d'
{
			"query": {
				"bool": {
					"should": [
						{"match":{"name":"andrew"}}
					],
					"must": [
					    {"match":{"birth_year":"2005"}}
					]
				}
			},

	        "sort":{"_score":{"order":"desc"}},"from":0,"size":"15","min_score":2,"score_mode":"max"
}
' | python -m json.tool

i get another error:

{
    "error": {
        "col": 86,
        "line": 14,
        "reason": "Unknown key for a VALUE_STRING in [score_mode].",
        "root_cause": [
            {
                "col": 86,
                "line": 14,
                "reason": "Unknown key for a VALUE_STRING in [score_mode].",
                "type": "parsing_exception"
            }
        ],
        "type": "parsing_exception"
    },
    "status": 400
}

So, something still wrong...
However, after i removed "min_score" from query, everything started to work.

Probably the problem because of wrong usage of "min_score"?

I've unfortunately run into this same problem... the query worked fine up to and including v6.8.0, but when we updated to v7.1.1 we began experiencing the NPEs at the exact place you are. We've tried moving back to the initial GA release 7.0.0 and still experience the same problem. If we remove the min score it bypasses the error, but having the min score is pretty important for us.

Ping @jimczi. Aware of this?

The NPE is indeed a bug. Can you open an issue on github?
Regarding the other error with score_mode, function_score and min_score are two different features so this is expected. score_mode is not applicable in the request body, it can only be used inside a function_score query.

I quickly and haphazardly filed https://github.com/elastic/elasticsearch/issues/43497

My mappings and queries are probably more convoluted than what @davin4u has provided, so I'm guessing it will be easier to work with his reproduction steps.

I am also having a NPE issue, but it is different. I came across this thread when trying to find a reason/solution. Searching for "TopScoreDocCollector" is what brought me here. Figured I would piggyback off of it, but can start a new thread if more appropriate. I am on elastic v7.2.0 and can provide more details if necessary.

My issue seems to be RBAC/ABAC related. As a superuser I have no issues. As a user that belongs to a role where documents must match a hard coded value, no issues. But then I have a user where {{_user.email}} must match a particular field in the documents. For that user, when I log into Kibana, everything mostly works, but I get a "x of y shards failed" message popup. I dug a little deeper and there are five specific indices that are part of a rolling index that are the ones that fail. The DEBUG error message repeats itself five times for those five indices. Here is one of those DEBUG error messages:

[I got an error message saying the body is over 7000 characters, so I am going to post DEBUG error in a second message. I used </>. Is there a better way to do this??]

You can see the NPE message in there. There is no mention of "MinimumScoreCollector" which I know is the context of the OPs issue. Please let me know if there are any additional details that may be of use.

[2019-07-10T16:46:10,508][DEBUG][o.e.a.s.TransportSearchAction] [node1] [data_2019.07.01][0], node[kYerohYBQ8-wr52Mbhvn1Q], [P], s[STARTED], a[id=dmTDB8InRR27JGPws1KNeA]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[data_2018.12.26, data_2018.12.25,..., data_2019.06.25], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=false, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={}}]
org.elasticsearch.transport.RemoteTransportException: [node2][10.1.1.2:9300][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
	at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:306) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:114) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:335) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:360) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:145) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.lang.NullPointerException
	at org.apache.lucene.search.TopScoreDocCollector$SimpleTopScoreDocCollector$1.collect(TopScoreDocCollector.java:68) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
	at org.elasticsearch.xpack.core.security.authz.accesscontrol.SecurityIndexSearcherWrapper.intersectScorerAndRoleBits(SecurityIndexSearcherWrapper.java:199) ~[?:?]
	at org.elasticsearch.xpack.core.security.authz.accesscontrol.SecurityIndexSearcherWrapper$IndexSearcherWrapper.search(SecurityIndexSearcherWrapper.java:169) ~[?:?]
	at org.apache.lucene.search.XIndexSearcher.search(XIndexSearcher.java:44) ~[elasticsearch-7.2.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
	at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:177) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:443) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
	at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:271) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:114) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:335) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:360) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:145) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]
1 Like

Hey James

Could you open an issue in GitHub with this stacktrace and if possible share the query and your settings?