Courier Fetch Warnings

Visualizations and dashboards working fine until today. I started seeing Courier fetch warnings. I see the following errors in the ES logs. I am not sure how to proceed.

RemoteTransportException[[N01][][indices:data/read/search[phase/query]]]; nested: SearchParseException[failed to parse search source [{"query":{"filtered":{"query":{"query_string":{"analyze_wildcard":true,"query":""}},"filter":{"bool":{"must":[{"query":{"query_string":{"analyze_wildcard":true,"query":""}}},{"range":{"@timestamp":{"gte":1487992829615,"lte":1488079229615,"format":"epoch_millis"}}}],"must_not":[]}}}},"size":0,"aggs":{"2":{"terms":{"field":"TargetUserName","size":5,"order":{"_count":"desc"}},"aggs":{"3":{"terms":{"field":"TargetDomainName","size":5,"order":{"_count":"desc"}}}}}}}]]; nested: IllegalStateException[Field data loading is forbidden on [TargetUserName]];
Caused by: SearchParseException[failed to parse search source [{"query":{"filtered":{"query":{"query_string":{"analyze_wildcard":true,"query":""}},"filter":{"bool":{"must":[{"query":{"query_string":{"analyze_wildcard":true,"query":""}}},{"range":{"@timestamp":{"gte":1487992829615,"lte":1488079229615,"format":"epoch_millis"}}}],"must_not":[]}}}},"size":0,"aggs":{"2":{"terms":{"field":"TargetUserName","size":5,"order":{"_count":"desc"}},"aggs":{"3":{"terms":{"field":"TargetDomainName","size":5,"order":{"_count":"desc"}}}}}}}]]; nested: IllegalStateException[Field data loading is forbidden on [TargetUserName]];
at org.elasticsearch.transport.TransportRequestHandler.messageReceived(
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(
at org.elasticsearch.transport.TransportService$4.doRun(
at java.util.concurrent.ThreadPoolExecutor.runWorker(
at java.util.concurrent.ThreadPoolExecutor$
Caused by: java.lang.IllegalStateException: Field data loading is forbidden on [TargetUserName]
at org.elasticsearch.index.fielddata.IndexFieldDataService$
at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(
... 12 more

Looks like it is telling me that "Field data loading is forbidden on [TargetUserName]". There are multiple entries of these referring to different fields. No prior issues until today with the data.

What's the mapping on the field in the log look like?

Hope this is what you are asking...

      "TargetUserName" : {
        "type" : "string",
        "index" : "not_analyzed"

Looks like there could be some issues with today's index. I went back to just searching for data from yesterday or two days ago, and I don't get the warnings.

Now the status is green but perhaps there is something wrong with the shards? how would I find out?

For example: here are the shards for today

index shard prirep state docs store ip node
events-eventlog-2017.02.26 1 r STARTED 58983 96.2mb N02
events-eventlog-2017.02.26 1 p STARTED 58940 96.5mb N01
events-eventlog-2017.02.26 3 p STARTED 58863 96.5mb N03
events-eventlog-2017.02.26 3 r STARTED 58863 96.2mb N01
events-eventlog-2017.02.26 2 r STARTED 59136 96.4mb N03
events-eventlog-2017.02.26 2 p STARTED 59136 96.4mb N02
events-eventlog-2017.02.26 4 p STARTED 58954 96.9mb N03
events-eventlog-2017.02.26 4 r STARTED 58970 96.9mb N01
events-eventlog-2017.02.26 0 r STARTED 58856 95.8mb N02
events-eventlog-2017.02.26 0 p STARTED 58925 95.9mb N01

Has the mapping changed between this date and previous ones?

You are right. Looks like there is new mapping.

      "TargetUserName" : {
        "type" : "string",
        "index" : "not_analyzed",
        "fielddata" : {
          "format" : "disabled"

what's the best approach to handle this? now that there's a difference. Alias?

That shouldn't cause a problem as they are both not_analyzed and fielddata is disabled accordingly.

What version are you on?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.