Show cumulative duration between start / stop events

Elastic Stack Kibana
1

Hello,

I am logging firewall events and see a process start and stop throughout the day. I want to show for how much of the day the process was running, by the cumulative sum of:

@timestamp (end_event) - @timestamp (start_event)

Is this possible in Kibana?

Thank you.

2

As you mentioned @timestamp, I'll assume you are using Logstash. Further, I'll assume your start and stop events are being stored in indices named logstash-* (where * is some time-based value)

Then you do something like this in your Logstash pipeline:

  • When you encounter an end event, use the elasticsearch filter plugin to query the logstash-* indices for the corresponding start event document.
  • Grab the start time from that start event document and use the end time from the end event document currently being processed to compute the difference = process lifetime.
  • Add a field for the process lifetime in the end event document. Say this field is called process_lifetime.

Then in Kibana you can visualize the end event documents, aggregating on the sum of process_lifetime.

2 Likes
3

Thanks for your reply.

Yes, I am using Logstash and you are correct about the indices, but to do what you suggest I would need to know in advance what I am looking for before I ingest the data. I'm ingesting the data into Elasticsearch for analysis, so I don't necessarily know what I'm looking for before I find an example.

There may be multiple instances where I want to display a cumulative time, and looking at one may give me cause to analyse another, but I can't re-ingest historical data via Logstash to find the duration of a different process.

4

Check out Elapsed Filter. It is just for this.

I have used it, I can give you a hand.

1 Like
5

Thank you for your reply. I will take a look. I'm building a new cluster at the moment so will test this when it's up.