ES crashes when parsing fails due to mapping failure

assaf · February 5, 2015, 12:56pm

We have automatic mapping turned for our logstash indexes. Every now and
then our system logs a record that has a "wrong" (out of the ordinary)
field data type.
For example, a field that's been automatically mapped to be a number
occasionally is logged as a string.
This causes ES to crash with the following stack trace:

at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NumberFormatException: For input string: "1.0"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:441)
at java.lang.Long.parseLong(Long.java:483)
at
org.elasticsearch.index.mapper.core.NumberFieldMapper.parseLongValue(NumberFieldMapper.java:357)
at
org.elasticsearch.index.mapper.core.LongFieldMapper.termQuery(LongFieldMapper.java:185)
at
org.apache.lucene.queryparser.classic.MapperQueryParser.getFieldQuerySingle(MapperQueryParser.java:257)
at
org.apache.lucene.queryparser.classic.MapperQueryParser.getFieldQuery(MapperQueryParser.java:168)
at
org.apache.lucene.queryparser.classic.QueryParserBase.getFieldQuery(QueryParserBase.java:487)
at
org.apache.lucene.queryparser.classic.MapperQueryParser.getFieldQuery(MapperQueryParser.java:287)
at
org.apache.lucene.queryparser.classic.QueryParserBase.handleQuotedTerm(QueryParserBase.java:875)
at
org.apache.lucene.queryparser.classic.QueryParser.Term(QueryParser.java:464)
at
org.apache.lucene.queryparser.classic.QueryParser.Clause(QueryParser.java:259)
at
org.apache.lucene.queryparser.classic.QueryParser.Query(QueryParser.java:183)
at
org.apache.lucene.queryparser.classic.QueryParser.Clause(QueryParser.java:263)
at
org.apache.lucene.queryparser.classic.QueryParser.Query(QueryParser.java:183)
at
org.apache.lucene.queryparser.classic.QueryParser.TopLevelQuery(QueryParser.java:172)
at
org.apache.lucene.queryparser.classic.QueryParserBase.parse(QueryParserBase.java:123)
at
org.apache.lucene.queryparser.classic.MapperQueryParser.parse(MapperQueryParser.java:882)
at
org.elasticsearch.index.query.QueryStringQueryParser.parse(QueryStringQueryParser.java:223)
at
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:277)
at
org.elasticsearch.index.query.FQueryFilterParser.parse(FQueryFilterParser.java:66)
at
org.elasticsearch.index.query.QueryParseContext.executeFilterParser(QueryParseContext.java:343)
at
org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:324)
at
org.elasticsearch.index.query.BoolFilterParser.parse(BoolFilterParser.java:92)
at
org.elasticsearch.index.query.QueryParseContext.executeFilterParser(QueryParseContext.java:343)
at
org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:324)
at
org.elasticsearch.index.query.FilteredQueryParser.parse(FilteredQueryParser.java:74)
at
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:277)
at
org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:382)
at
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
at
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:276)
at
org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)

Is there a way to tell ES not to crash when failing to parse a field? I
realize I can override the mapping, and so on, but regardless I'm also
interested in getting ES to run reliably without crashing on rare inputs.

Thanks,
Assaf

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9b6febf3-5800-40a6-8cf4-fcb24ac8abed%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Itamar_Syn_Hershko · February 5, 2015, 1:03pm

It's not crashing, it is just a log that says the document insert was
rejected

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 5, 2015 at 2:56 PM, assaf@singular.net wrote:

We have automatic mapping turned for our logstash indexes. Every now and
then our system logs a record that has a "wrong" (out of the ordinary)
field data type.
For example, a field that's been automatically mapped to be a number
occasionally is logged as a string.
This causes ES to crash with the following stack trace:

at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NumberFormatException: For input string: "1.0"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:441)
at java.lang.Long.parseLong(Long.java:483)
at
org.elasticsearch.index.mapper.core.NumberFieldMapper.parseLongValue(NumberFieldMapper.java:357)
at
org.elasticsearch.index.mapper.core.LongFieldMapper.termQuery(LongFieldMapper.java:185)
at
org.apache.lucene.queryparser.classic.MapperQueryParser.getFieldQuerySingle(MapperQueryParser.java:257)
at
org.apache.lucene.queryparser.classic.MapperQueryParser.getFieldQuery(MapperQueryParser.java:168)
at
org.apache.lucene.queryparser.classic.QueryParserBase.getFieldQuery(QueryParserBase.java:487)
at
org.apache.lucene.queryparser.classic.MapperQueryParser.getFieldQuery(MapperQueryParser.java:287)
at
org.apache.lucene.queryparser.classic.QueryParserBase.handleQuotedTerm(QueryParserBase.java:875)
at
org.apache.lucene.queryparser.classic.QueryParser.Term(QueryParser.java:464)
at
org.apache.lucene.queryparser.classic.QueryParser.Clause(QueryParser.java:259)
at
org.apache.lucene.queryparser.classic.QueryParser.Query(QueryParser.java:183)
at
org.apache.lucene.queryparser.classic.QueryParser.Clause(QueryParser.java:263)
at
org.apache.lucene.queryparser.classic.QueryParser.Query(QueryParser.java:183)
at
org.apache.lucene.queryparser.classic.QueryParser.TopLevelQuery(QueryParser.java:172)
at
org.apache.lucene.queryparser.classic.QueryParserBase.parse(QueryParserBase.java:123)
at
org.apache.lucene.queryparser.classic.MapperQueryParser.parse(MapperQueryParser.java:882)
at
org.elasticsearch.index.query.QueryStringQueryParser.parse(QueryStringQueryParser.java:223)
at
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:277)
at
org.elasticsearch.index.query.FQueryFilterParser.parse(FQueryFilterParser.java:66)
at
org.elasticsearch.index.query.QueryParseContext.executeFilterParser(QueryParseContext.java:343)
at
org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:324)
at
org.elasticsearch.index.query.BoolFilterParser.parse(BoolFilterParser.java:92)
at
org.elasticsearch.index.query.QueryParseContext.executeFilterParser(QueryParseContext.java:343)
at
org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:324)
at
org.elasticsearch.index.query.FilteredQueryParser.parse(FilteredQueryParser.java:74)
at
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:277)
at
org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:382)
at
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
at
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:276)
at
org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)

Is there a way to tell ES not to crash when failing to parse a field? I
realize I can override the mapping, and so on, but regardless I'm also
interested in getting ES to run reliably without crashing on rare inputs.

Thanks,
Assaf

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9b6febf3-5800-40a6-8cf4-fcb24ac8abed%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/9b6febf3-5800-40a6-8cf4-fcb24ac8abed%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4Ztvpc%2BgCDB_ZGcquLnpxETPuOicovL8UKozhm9_cNpYew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.