Timelion showing query result search error

Elastic Stack Kibana
1

I have used timelion to do some query and make some time series data.
This is my query
.es(index='linux_cpu-*', metric='avg:CPU(%)', split='Hostname:5')
I didn't find anything wrong with this query but just could not understand why it could not generate the desired result. Then, when I cross check with elasticsearch data log, I found the following stating search parse error.

[2016-06-26 23:32:43,290][DEBUG][action.search            ] [Bentley Wittman] [linux_cpu-2015.01.21][4], node[rtIRvgffRha3nFqYhjh8YA], [P], v[2], s[STARTED], a[id=xIKFCsvnRKaGqt7XyJtHNg]: Failed to execute [org.elasticsearch.action.search.SearchRequest@148c67f] lastShard [true]
RemoteTransportException[[Bentley Wittman][172.16.1.238:9300][indices:data/read/search[phase/query]]]; nested: SearchParseException[failed to parse search source [{"query":{"bool":{"must":[{"range":{"@timestamp":{"gte":1420096382918,"lte":1422689282919,"format":"epoch_millis"}}}],"must_not":[],"filter":{}}},"aggs":{"q":{"meta":{"type":"split"},"filters":{"filters":{"*":{"query_string":{"query":"*"}}}},"aggs":{"Hostname":{"meta":{"type":"split"},"terms":{"field":"Hostname","size":5},"aggs":{"time_buckets":{"meta":{"type":"time_buckets"},"date_histogram":{"field":"@timestamp","interval":"1w","time_zone":"Asia/Shanghai","extended_bounds":{"min":1420096382918,"max":1422689282919},"min_doc_count":0},"aggs":{"avg(CPU(%))":{"avg":{"field":"CPU(%)"}}}}}}}}},"size":0}]]; nested: IllegalStateException[Field data loading is forbidden on [Hostname]];
Caused by: SearchParseException[failed to parse search source [{"query":{"bool":{"must":[{"range":{"@timestamp":{"gte":1420096382918,"lte":1422689282919,"format":"epoch_millis"}}}],"must_not":[],"filter":{}}},"aggs":{"q":{"meta":{"type":"split"},"filters":{"filters":{"*":{"query_string":{"query":"*"}}}},"aggs":{"Hostname":{"meta":{"type":"split"},"terms":{"field":"Hostname","size":5},"aggs":{"time_buckets":{"meta":{"type":"time_buckets"},"date_histogram":{"field":"@timestamp","interval":"1w","time_zone":"Asia/Shanghai","extended_bounds":{"min":1420096382918,"max":1422689282919},"min_doc_count":0},"aggs":{"avg(CPU(%))":{"avg":{"field":"CPU(%)"}}}}}}}}},"size":0}]]; nested: IllegalStateException[Field data loading is forbidden on [Hostname]];
	at org.elasticsearch.search.SearchService.parseSource(SearchService.java:855)
	at org.elasticsearch.search.SearchService.createContext(SearchService.java:654)
	at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:620)
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:371)
	at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
	at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
	at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Field data loading is forbidden on [Hostname]
	at org.elasticsearch.index.fielddata.IndexFieldDataService$1.build(IndexFieldDataService.java:74)
	at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:275)
	at org.elasticsearch.search.aggregations.support.ValuesSourceParser.config(ValuesSourceParser.java:209)
	at org.elasticsearch.search.aggregations.bucket.terms.TermsParser.parse(TermsParser.java:76)
	at org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:198)
	at org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:176)
	at org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:103)
	at org.elasticsearch.search.aggregations.AggregationParseElement.parse(AggregationParseElement.java:60)
	at org.elasticsearch.search.SearchService.parseSource(SearchService.java:838)
	... 12 more

Does anyone know what does that mean and how should I solve it?

2

You should use split='Hostname.raw:5' instead.

his is not a bug. It is a safeguard. The logstash template now disables fielddata loading where it makes sense, eg see https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/elasticsearch-template.json#L15

You get this message when you try to sort or run aggregations or scripts on analyzed fields. Fulfilling this request would cause massive amounts of memory usage on your cluster, and it almost certainly isn't what you want anyway, eg Field data loading is forbidden on path"... You don't want to aggregate on the analyzed field path, you want to aggregate on the not analyzed field path.raw, which uses doc values not heap memory.

3

Do you still have an issue or did using the .raw resolve it?

5

Yes. I have partly solved it. When I use raw field, and I tried to use with moving average function together, the result is not properly displayed, or in other words, does not make any senses. For a simple testing. I have made the following query.

.es(index='cpu-*', metric='avg:CPU(%)', q='Hostname.raw:tested3a').movingaverage(20)

Error.PNG1247×567 19.2 KB

But as can seen from the chart, the line will eventually become 0 for that 20 days of moving average which does not make any senses. What could be the potential problem?