I am logging firewall events and see a process start and stop throughout the day. I want to show for how much of the day the process was running, by the cumulative sum of:
@timestamp (end_event) - @timestamp (start_event)
Is this possible in Kibana?
As you mentioned
@timestamp, I'll assume you are using Logstash. Further, I'll assume your start and stop events are being stored in indices named
* is some time-based value)
Then you do something like this in your Logstash pipeline:
- When you encounter an end event, use the elasticsearch filter plugin to query the
logstash-* indices for the corresponding start event document.
- Grab the start time from that start event document and use the end time from the end event document currently being processed to compute the difference = process lifetime.
- Add a field for the process lifetime in the end event document. Say this field is called
Then in Kibana you can visualize the end event documents, aggregating on the sum of
Thanks for your reply.
Yes, I am using Logstash and you are correct about the indices, but to do what you suggest I would need to know in advance what I am looking for before I ingest the data. I'm ingesting the data into Elasticsearch for analysis, so I don't necessarily know what I'm looking for before I find an example.
There may be multiple instances where I want to display a cumulative time, and looking at one may give me cause to analyse another, but I can't re-ingest historical data via Logstash to find the duration of a different process.
Check out Elapsed Filter. It is just for this.
I have used it, I can give you a hand.
Thank you for your reply. I will take a look. I'm building a new cluster at the moment so will test this when it's up.