2D Heatmap for startTime

[guyasp (Guy Ashiri-Prossner)]

Hello,
I have a logging of records with creation timestamps, named startTime. I would like to create a 2D heatmap showing peak hours - that is, my horizontal axis should be set to "Date histogram" of this values, with minimal interval of 1 day. That's done of course.
However, I fail to create the vertical axis, which should be the hour of the day (could be rounded to the hour, no problem). That is, I need to extract the hour part from the timestamps, and then create the counting bins as an "outer product" of the dates in the range and the 24 hours.

Is this something achievable?

[stephenb (Stephen Brown)]

Hi @guyasp Welcome to the community!

@guyasp

Is this something achievable?

Yes absolutely but you are going to need to do a little work...

What version are you on?

There are 2 approaches ...

1. On Ingest parse out the hour_of_day through and ingest pipeline, then that data would be available for many uses and this the visualization would be fairly trivial you could use an ingest pipeline

2. Create a Runtime Field ... not as good for scale and speed but could test that out then decide

ZonedDateTime zdt = ZonedDateTime.parse(doc['@timestamp'].value.toString());
emit(zdt.getHour());

I created a runtime fields on the data view

The Heat Map (filter down to logs you are interested in)

Horizontal Date Histogram on @timestamp
Cell Value count
Vertical access hour_of_day

And I got this...

Runtime fields are OK to use but they do not scale really well if you were going to do this for large data sets I would parse out the hour in an ingest pipeline so it is an indexed field

[guyasp (Guy Ashiri-Prossner)]

Thanks!