Hi,
I was observing some "Courier fetch: shards failed exceptions" in Kibana. I was finally able to root-cause it due to the scripted fields.
Unfortunately, there was no information given in the Elasticsearch logs by default.
I could see the exception only when I enabled TRACE level logging for "logger.org.elasticsearch.transport".
Shouldn't this exception be at WARNING level at least?
Here's the complete exception:
[2018-09-17T08:54:51,720][TRACE][o.e.t.T.tracer ] [es-docker-data1] [29690526][indices:data/read/search[phase/query]] sent error response
java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [analysis_type] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
at org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:518) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:115) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.index.query.QueryShardContext.getForField(QueryShardContext.java:165) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.resolve(ValuesSourceConfig.java:96) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.resolveConfig(ValuesSourceAggregationBuilder.java:318) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:311) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:38) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:139) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:329) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:766) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.SearchService.createContext(SearchService.java:575) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:551) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:347) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:333) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:329) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1019) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:725) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.3.1.jar:6.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:844) [?:?]