INTERNAL_SERVER_ERROR null_pointer_exception while percolating a document

Elastic Stack Elasticsearch
1

We have an index with 10 shards on 10 nodes, and sometimes when a document is percolated, some nodes fail with a null_pointer_exception. A second try does not reproduce the exception.

status: 'INTERNAL_SERVER_ERROR',
reason: { type: 'null_pointer_exception', reason: null }

We use ES 2.4.1, but could not find a bug fix relating to this in later 2.4.* releases. Did we miss it? Was it fixed in a 5.* release?

We cannot try ES 5.* because of incompatibilities in our code base.

I found a thread there with a similar issue but no answers.

2

Hey @jazz do you see a stacktrace in your log files that related to percolating?
If so can you share these stack traces?

I was able to reproduce the issue in the other issue that you've mentioned. Would be good to know if whether you have encountered the same error.

3

I could not see a related stack trace. Do I have to enable a setting?

4

Maybe try setting the action.percolate to debug or trace level in logging.yml or update the log settings in realtime via the update cluster setting api: https://www.elastic.co/guide/en/elasticsearch/guide/current/logging.html ?

5

@jazz I commented on the issue you mentioned with a likely cause for this issue that you're encountering.

6

With the setting in debug, I got this stacktrace:

RemoteTransportException[[es16-elasticsearch_1][172.17.0.19:9300][indices:data/read/percolate[s]]]; nested: PercolateException[failed to percolate]; nested: PercolateException[failed to execute]; nested: NullPointerException;
Caused by: PercolateException[failed to percolate]; nested: PercolateException[failed to execute]; nested: NullPointerException;
	at org.elasticsearch.action.percolate.TransportPercolateAction.shardOperation(TransportPercolateAction.java:181)
	at org.elasticsearch.action.percolate.TransportPercolateAction.shardOperation(TransportPercolateAction.java:56)
	at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$ShardTransportHandler.messageReceived(TransportBroadcastAction.java:282)
	at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$ShardTransportHandler.messageReceived(TransportBroadcastAction.java:278)
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
	at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:293)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: PercolateException[failed to execute]; nested: NullPointerException;
	at org.elasticsearch.percolator.PercolatorService$5.doPercolate(PercolatorService.java:620)
	at org.elasticsearch.percolator.PercolatorService.percolate(PercolatorService.java:259)
	at org.elasticsearch.action.percolate.TransportPercolateAction.shardOperation(TransportPercolateAction.java:178)
	... 9 more
Caused by: java.lang.NullPointerException
	at org.apache.lucene.spatial.geopoint.search.GeoPointTermQueryConstantScoreWrapper$1$1.matches(GeoPointTermQueryConstantScoreWrapper.java:132)
	at org.apache.lucene.search.ConjunctionDISI$TwoPhaseConjunctionDISI.matches(ConjunctionDISI.java:248)
	at org.apache.lucene.search.ConjunctionDISI$TwoPhaseConjunctionDISI.matches(ConjunctionDISI.java:248)
	at org.apache.lucene.search.ConjunctionDISI$TwoPhase.matches(ConjunctionDISI.java:291)
	at org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:184)
	at org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:207)
	at org.elasticsearch.common.lucene.Lucene.exists(Lucene.java:254)
	at org.elasticsearch.percolator.QueryCollector$Match.collect(QueryCollector.java:184)
	at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:221)
	at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
	at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:821)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)
	at org.elasticsearch.percolator.PercolatorService.queryBasedPercolating(PercolatorService.java:818)
	at org.elasticsearch.percolator.PercolatorService.access$400(PercolatorService.java:108)
	at org.elasticsearch.percolator.PercolatorService$5.doPercolate(PercolatorService.java:611)
	... 11 more
7

Thanks for that stacktrace. This is a different failure and has to do with the fact that the in memory index that the percolator is using in 2.x doesn't support doc values (column stride storage that some geo queries use). In 5.x and beyond that percolator does support doc values.

There is a workaround that you can try to do, setting the create index version of your index to 2.1 or before as is mentioned here: https://github.com/elastic/elasticsearch/issues/16832#issuecomment-197329289

8

Do I have to re-create the index with the version set to 2.1 or can I change the version of the index?

9

Hum, we have many regular geo queries on the index that hold the percolators, it does not seem to be a good idea to change the version of this index performance-wise.

It seems like we have two alternatives:

  • migrate to ES 5.x
  • create a second index with same documents as the first one, but with the index version set to 2.1, where the percolators will be stored. Not sure if it will good for our cluster with twice the data.
10

Do I have to re-create the index with the version set to 2.1 or can I change the version of the index?

What you can try is close the index and then adding the index created settings via the update index settings api. I think it works, but I'm not 100% sure.

Hum, we have many regular geo queries on the index that hold the percolators, it does not seem to be a good idea to change the version of this index performance-wise.

These improvements added in later 2.x versions are only applicable if these geo queries run on a real index. Not in the case of the percolator, because the geo queries are executed on a in memory index holding a single document.

11

The issue is that we have lots of geo queries on this index beside the percolators.

12

The issue is that we have lots of geo queries on this index beside the percolators.

Then maybe split the percolator queries from the real data in a dedicated index? If you do that then make sure that you that you also copy the mapping for the documents you percolate into the dedicated index.

13

I missed the fact that I can have the percolator queries in a dedicated index! I will try this, thanks for your help!

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.