ElasticSearch stops responding after reaching a certain number of "fails to parse query" errors

Elastic Stack Elasticsearch
4

We have the logs from multiple instances where this error happened. I don't see an option to share the files. If you can provide your email address i can send them over.

This has been happening for quite a while now and every time the service goes in to this state, the logs always had this query error as the last item. We never thought a parse error in the query could cause this and we ran several unit tests to simulate the same but the issue never happened during those tests. Only when we have run this a certain number of times, this issue happens.

Here is a sample of the query,

"must" : [
{
"query_string" : {
"query" : "sam\",
"fields" : [
"data.contexts.all_values^1.0"
],
"use_dis_max" : true,
"tie_breaker" : 0.0,
"default_operator" : "and",
"auto_generate_phrase_queries" : false,
"max_determined_states" : 10000,
"enable_position_increment" : true,
"fuzziness" : "AUTO",
"fuzzy_prefix_length" : 0,
"fuzzy_max_expansions" : 50,
"phrase_slop" : 0,
"escape" : false,
"split_on_whitespace" : true,
"boost" : 1.0
}
}
],

The last logs,

org.elasticsearch.transport.RemoteTransportException: [VD6DLR4][10.0.0.63:9300][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.index.query.QueryShardException: Failed to parse query [sam]
at org.elasticsearch.index.query.QueryStringQueryBuilder.doToQuery(QueryStringQueryBuilder.java:1034) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:97) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:442) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:416) ~[elasticsearch-5.2.0.jar:5.2.0]

Caused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: parse_exception: Cannot parse 'sam': Lexical error at line 1, column 5. Encountered: after : ""
at org.apache.lucene.queryparser.classic.QueryParserBase.parse(QueryParserBase.java:116) ~[lucene-queryparser-6.4.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:58:02]
at org.apache.lucene.queryparser.classic.MapperQueryParser.parse(MapperQueryParser.java:786) ~[elasticsearch-5.2.0.jar:6.4.0 bbe4b08cc1fb673d0c3eb4b8455f23ddc1364124 - jim - 2017-01-17 15:58:02]
at org.elasticsearch.index.query.QueryStringQueryBuilder.doToQuery(QueryStringQueryBuilder.java:1032) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:97) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:442) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:416) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:97) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.index.query.NestedQueryBuilder.doToQuery(NestedQueryBuilder.java:252) ~[elasticsearch-5.2.0.jar:5.2.0]