Durations and counts between events

Greetings Elastic Stack experts.

I have an index of timestamped events like so (which I have transformed into JSON, but using simple strings here for readability):

ID: 1, name: ABC, time: 111.11, action: start
ID: 2, name: DEF, time: 112.0, action: start
ID: 1, name: , time: 114, action: join
ID: 1, name: ABC, time: 123, action: end

ID: 1, name: ABC, time: 249, action: start
ID: 7, name: XYZ, time: 250.0, action: start
ID: 1, name: , time: 260, action: end

I need to compute elapsed times and join counts for each occurrence of ID: 1, and so on. A summary along these lines would be even better:

ID: 1, name: ABC, start_times: [111.11, 249], end_times: [123, 260], join_times: [114]

Is it possible to do this in Kibana, maybe using a script?

The logstash elapsed filter did not give me I did try the elapsed filter in Logstash but it resulted in some errors due to which I had to manually validate a whole bunch of data.

Any guidance?


You would likely need to use a scripted metric aggregation, and then use the Vega visualization to run that scripted aggregation and display how you would like. I will warn, you are essentially going to be building the aggregation and the visualization - aka, this will not be easy.

Thanks Jared. Can you suggest an alternate? Maybe filters in Logstash?

The logstash-filter-aggregate provides some good examples that fit in with your scenario. It does come with some limitations though:

You should be very careful to set Logstash filter workers to 1 ( -w 1 flag) for this filter to work correctly otherwise events may be processed out of sequence and unexpected results will occur.


If you know that this type of data will come from a single source, you can isolate it to a dedicated logstash instance (or a logstash pipeline with the pipeline.workers set to 1. Theoretically should work, but I've never actually tried with the aggregate filter)

Thanks Jared. I'll try it out.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.