Calculating duration between two documents in Elasticsearch


(Tim Arp) #1

Hi,

I'm collecting logs from an application recording the start time. Then another record is signifying the end time. Both records have to match on DeviceId and a CorrelationId. I'm using an ingest pipeline to process the logs. When the log line is processed that is the end time, I want to query ES and get the starttime and calculate the difference, and add a field for it.
I'm at a loss on how and if this can be done in pipelines in Elasticsearch.
I saw something similar via Logstash, but I'm trying to avoid Logstash and just do this in Elasticsearch.

Any ideas?
Thanks,
Tim


(Mark Harwood) #2

Use an entity centric index - see https://youtu.be/yBf7oeJKH2Y

The comments include a link to example data and scripts


Calculate time difference between two different record timestamps
(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.


(Mark Harwood) #4

No they don't because I expect they are awaiting a review which is why only I can see them.

I've just updated the scripts to work with elasticsearch 6.3 and this is the link