SearchParseException[failed to parse search source [_na_]];

Elastic Stack Elasticsearch
1

Could you point please me in the right direction. I have a small cluster with 2 nodes [MX203][10.40.0.203:9300] and [MX105][10.40.0.105:9300] . Periodically logs are overwhelmed with tons of messages like

[2016-07-28 12:08:48,220][DEBUG][action.search ] [MX203] [indexname][0], node[MqfyLJodStKEdgTyaIVbiw], [R], v[31], s [STARTED], a[id=DUDfV75ETbmtOjzsbijXiA]: Failed to execute [org.elasticsearch.action.search.SearchRequest@57d98094] lastShard [true] RemoteTransportException[[MX105][10.40.0.105:9300][indices:data/read/search[phase/query]]]; nested: SearchParseException[failed to parse search source [_na_]]; nested: ElasticsearchParseException[Failed to derive xcontent]; Caused by: SearchParseException[failed to parse search source [_na_]]; nested: ElasticsearchParseException[Failed to derive xcontent]; at org.elasticsearch.search.SearchService.parseSource(SearchService.java:855) at org.elasticsearch.search.SearchService.createContext(SearchService.java:654) at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:620) at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:371) at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java: 368) at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java: 365) at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:300) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: ElasticsearchParseException[Failed to derive xcontent] at org.elasticsearch.common.xcontent.XContentFactory.xContent(XContentFactory.java:240) at org.elasticsearch.search.SearchService.parseSource(SearchService.java:824) ... 12 more

where [indexname] is any of existing indexes even not participating in search.

I'm confused with :9300, so it look like replication? But this is definitely connected to queries as if queries are stopped, exceptions do not appear in log anymore.

Cluster is green.
Elasticsearch 2.3.4 on Ubuntu

2

That's a query that cannot be parsed on index indexname, shard 0.

Maybe some malformed query ends up being executed against that index? The log you see contains an error received from another node. The nodes communication through the transport layer, and that is why you see the 9300 port in there.

I'd suggest to verify which queries get sent to the cluster, and double check whether you get back any error. Also, do check the _shards header that the search api returns. That section contains how many shards successfully executed the query, and how many failed. It can happen that you get a 200 - OK response although some shards failed.

3

Thanks. I'm understanding better how things work now .

We found one bad written query raising those errors.