Calculating duration between two documents in Elasticsearch

Hi,

I'm collecting logs from an application recording the start time. Then another record is signifying the end time. Both records have to match on DeviceId and a CorrelationId. I'm using an ingest pipeline to process the logs. When the log line is processed that is the end time, I want to query ES and get the starttime and calculate the difference, and add a field for it.
I'm at a loss on how and if this can be done in pipelines in Elasticsearch.
I saw something similar via Logstash, but I'm trying to avoid Logstash and just do this in Elasticsearch.

Any ideas?
Thanks,
Tim

1 Like

Use an entity centric index - see https://youtu.be/yBf7oeJKH2Y

The comments include a link to example data and scripts

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

No they don't because I expect they are awaiting a review which is why only I can see them.

I've just updated the scripts to work with elasticsearch 6.3 and this is the link