Why is Kibana unable to aggregate non-indexed fields that have doc_values=True?

[leom (Leo)]

I have an index with a field, F, that has the following mapping:
"mapping": {
"type": "double",
"doc_values": True,
"index": False
}

Just as written in the ES guide, this should improve the performance of aggregation queries (like computing the average over the values of this field in some subset of the documents).

However, while I can perform aggregation queries by hand (e.g. in the Dev Tools console), I am unable to make plot F as a metric on the Y-axis in a Kibana visualization (like having a date histogram on a timestamp as the x-axis, and the average value of F per date bucket on the y-axis). Kibana complains that "No Compatible Fields: The "trades-*" index pattern does not contain any of the following field types: number".

Going to Management > Index Patterns, i see that the "trades-*" index has F as type number, but it does not have a check mark for "aggregatable". This seems internally consistent with Kibana's inability to visualize F as an aggregated metric. Yet, it makes no sense to me why Kibana claims it cannot aggregate over F, when I can in fact perform an aggregation query on it by hand.

the query:
GET /trades-*/_search
{
"size": 0,
"aggs": {
"test": {
"avg": {
"field": "F"
}
}
}
}

the result:
{
"took": 5,
"timed_out": false,
"_shards": {
"total": 53,
"successful": 53,
"failed": 0
},
"hits": {
"total": 334629,
"max_score": 0,
"hits": []
},
"aggregations": {
"test": {
"value": 5123.30
}
}
}

So, the question in the topic header: Why is Kibana unable to aggregate non-indexed fields that have doc_values=True?

Edit: I should add that I'm running Kibana 5.0.0 and Elasticsearch 5.0.0

[spalger (Spencer Alger)]

We moved to using the field_stats API in Kibana 5, and it provides the "aggregatable" and "searchable" attributes about fields. It seems though that turning off indexing for a field prevents it from showing up in the field stats response, so kibana assumes that the field is neither aggregatable or searchable.

I think this is a bug, and I've filled an issue with elasticsearch regarding it: https://github.com/elastic/elasticsearch/issues/21952

In the meantime, if you set "index" back to true then things should start working again.

[leom (Leo)]

Thank you Spencer!

Unfortunately, for my use case, I have disabled the _source field, so I will be unable to reindex the documents in a way that would easily get index: true again. Disabling source reduces the storage size by close to 50%, which would be a brutally high cost to pay for a bug (in addition to the time it takes to change the mapping and re-insert the > 1TB of data in the relevant indexes). I can do without the visualization for now, but will be keeping a close watch on the bug report.

[leom (Leo)]

It appears the change that fixes this issue will be released in the coming months, with version 5.2.0
(As per the github issue: https://github.com/elastic/elasticsearch/issues/21952)

[leom (Leo)]

I found a little hack that can make this work.

Let's say you want indexes A1, A2, ... AN to not index some fields that store metrics (but have doc_values: True). You can do the following to be able to visualize A1 ... AN.

1. Create an index A0 with a mapping that does index the fields. Don't insert any documents into it, though.
2. Add the index pattern A* to kibana. It should see A0, and will decide that the fields are able to be visualized.
3. Create A1 ... AN and insert your documents into them
4. You can now visualize A1 ... AN!