Durations and counts between events

Elastic Stack Kibana
1

Greetings Elastic Stack experts.

I have an index of timestamped events like so (which I have transformed into JSON, but using simple strings here for readability):

ID: 1, name: ABC, time: 111.11, action: start
ID: 2, name: DEF, time: 112.0, action: start
ID: 1, name: , time: 114, action: join
ID: 1, name: ABC, time: 123, action: end

ID: 1, name: ABC, time: 249, action: start
ID: 7, name: XYZ, time: 250.0, action: start
ID: 1, name: , time: 260, action: end

I need to compute elapsed times and join counts for each occurrence of ID: 1, and so on. A summary along these lines would be even better:

ID: 1, name: ABC, start_times: [111.11, 249], end_times: [123, 260], join_times: [114]

Is it possible to do this in Kibana, maybe using a script?

The logstash elapsed filter did not give me I did try the elapsed filter in Logstash but it resulted in some errors due to which I had to manually validate a whole bunch of data.

Any guidance?

Thanks

2

You would likely need to use a scripted metric aggregation, and then use the Vega visualization to run that scripted aggregation and display how you would like. I will warn, you are essentially going to be building the aggregation and the visualization - aka, this will not be easy.

3

Thanks Jared. Can you suggest an alternate? Maybe filters in Logstash?

4

The logstash-filter-aggregate provides some good examples that fit in with your scenario. It does come with some limitations though:

You should be very careful to set Logstash filter workers to 1 ( -w 1 flag) for this filter to work correctly otherwise events may be processed out of sequence and unexpected results will occur.

If you know that this type of data will come from a single source, you can isolate it to a dedicated logstash instance (or a logstash pipeline with the pipeline.workers set to 1. Theoretically should work, but I've never actually tried with the aggregate filter)

5

Thanks Jared. I'll try it out.