Courier Fetch Warnings

Elastic Stack Elasticsearch
1

Visualizations and dashboards working fine until today. I started seeing Courier fetch warnings. I see the following errors in the ES logs. I am not sure how to proceed.

RemoteTransportException[[N01][192.168.1.100:9300][indices:data/read/search[phase/query]]]; nested: SearchParseException[failed to parse search source [{"query":{"filtered":{"query":{"query_string":{"analyze_wildcard":true,"query":""}},"filter":{"bool":{"must":[{"query":{"query_string":{"analyze_wildcard":true,"query":""}}},{"range":{"@timestamp":{"gte":1487992829615,"lte":1488079229615,"format":"epoch_millis"}}}],"must_not":[]}}}},"size":0,"aggs":{"2":{"terms":{"field":"TargetUserName","size":5,"order":{"_count":"desc"}},"aggs":{"3":{"terms":{"field":"TargetDomainName","size":5,"order":{"_count":"desc"}}}}}}}]]; nested: IllegalStateException[Field data loading is forbidden on [TargetUserName]];
Caused by: SearchParseException[failed to parse search source [{"query":{"filtered":{"query":{"query_string":{"analyze_wildcard":true,"query":""}},"filter":{"bool":{"must":[{"query":{"query_string":{"analyze_wildcard":true,"query":""}}},{"range":{"@timestamp":{"gte":1487992829615,"lte":1488079229615,"format":"epoch_millis"}}}],"must_not":[]}}}},"size":0,"aggs":{"2":{"terms":{"field":"TargetUserName","size":5,"order":{"_count":"desc"}},"aggs":{"3":{"terms":{"field":"TargetDomainName","size":5,"order":{"_count":"desc"}}}}}}}]]; nested: IllegalStateException[Field data loading is forbidden on [TargetUserName]];
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:873)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:667)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:633)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:377)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Field data loading is forbidden on [TargetUserName]
at org.elasticsearch.index.fielddata.IndexFieldDataService$1.build(IndexFieldDataService.java:74)
at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:275)
at org.elasticsearch.search.aggregations.support.ValuesSourceParser.config(ValuesSourceParser.java:209)
at org.elasticsearch.search.aggregations.bucket.terms.TermsParser.parse(TermsParser.java:75)
at org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:198)
at org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:103)
at org.elasticsearch.search.aggregations.AggregationParseElement.parse(AggregationParseElement.java:60)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:856)
... 12 more

Looks like it is telling me that "Field data loading is forbidden on [TargetUserName]". There are multiple entries of these referring to different fields. No prior issues until today with the data.

2

What's the mapping on the field in the log look like?

3

Hope this is what you are asking...

      "TargetUserName" : {
        "type" : "string",
        "index" : "not_analyzed"
4

Looks like there could be some issues with today's index. I went back to just searching for data from yesterday or two days ago, and I don't get the warnings.

Now the status is green but perhaps there is something wrong with the shards? how would I find out?

For example: here are the shards for today

index shard prirep state docs store ip node
events-eventlog-2017.02.26 1 r STARTED 58983 96.2mb 192.168.1.101 N02
events-eventlog-2017.02.26 1 p STARTED 58940 96.5mb 192.168.1.100 N01
events-eventlog-2017.02.26 3 p STARTED 58863 96.5mb 192.168.1.102 N03
events-eventlog-2017.02.26 3 r STARTED 58863 96.2mb 192.168.1.100 N01
events-eventlog-2017.02.26 2 r STARTED 59136 96.4mb 192.168.1.102 N03
events-eventlog-2017.02.26 2 p STARTED 59136 96.4mb 192.168.1.101 N02
events-eventlog-2017.02.26 4 p STARTED 58954 96.9mb 192.168.1.102 N03
events-eventlog-2017.02.26 4 r STARTED 58970 96.9mb 192.168.1.100 N01
events-eventlog-2017.02.26 0 r STARTED 58856 95.8mb 192.168.1.101 N02
events-eventlog-2017.02.26 0 p STARTED 58925 95.9mb 192.168.1.100 N01

5

Has the mapping changed between this date and previous ones?

6

You are right. Looks like there is new mapping.

      "TargetUserName" : {
        "type" : "string",
        "index" : "not_analyzed",
        "fielddata" : {
          "format" : "disabled"
        }
      },

what's the best approach to handle this? now that there's a difference. Alias?

7

That shouldn't cause a problem as they are both not_analyzed and fielddata is disabled accordingly.

What version are you on?

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.