Generate Heatmap from histogram data

[PhracturedBlue]

I have data in elasticsearch in the form:
{ @timestamp: Jan 27, 2020 @ 05:57:44.837, aggregate: { values: [5, 10, 20, 50, 100], counts: [0, 5, 3, 8, 16]},
{ @timestamp: Jan 27, 2020 @ 05:47:44.829, aggregate: { values: [5, 10, 20, 50, 100], counts: [13, 6, 1, 0, 11]},

I am trying to figure out how to get Kibana to graph this data in a heatmap with the x-axis containing the timestamp and the y axis containing the buckets (5, 1,0, 15, 20, 50, 100). The value would then be the sum of all values in the given timestamp range for a given bucket index.

I can generate the proper x/y axes by using 'Terms' aggregation on the Y-axes and Date-Histogram aggregation on the x-asis. However, I cannot seem to get each box in the heatmap to contain the expected values. Instead The entire column has the same values

I've tried lots of different configurations, and can't seem to get anything that works.

[Marius_Dragomir]

Hi,
The Heatmap doesn't work with the way you have your data structured. Your data is already aggregated, it needs to be raw data or something like this:
{
timestamp: ...
aggregate.values = 5
aggregate.counts = 0
}

{
timestamp: ...
aggregate.values = 10
aggregate.counts = 5
}
and so on.

Then your settings for the heatmap will display the values that you actually want.

[PhracturedBlue]

Thank you for your response. It sounds like using Kibana won't be the best way for me to manage my graphs as all of my data is pre-aggregated, and duplicating the timestamps would be complicated (I have dozens of histogram-like fields stored per timestamp).

I see that elasticsearch 7.6 will support a native histogram model. Are there any plans to enable the heatmap with that model?

[Marius_Dragomir]

I don't see anything about that on the roadmap, but usually Kibana will be 1 or 2 versions behind ES regarding the adoptions of new features. So 7.7.0 or 8.0 sound more likely. You can also add an enhancement request in the Kibana repo about this and that will be the best way to follow development on the feature.

[wylie (Wylie Conlon)]

We are tracking this for Kibana here: https://github.com/elastic/kibana/issues/52426

[PhracturedBlue]

Thanks. I had opened a new request, but I've closed it and referenced that one. I'll monitor it for progress in the future.