Calculating duration between two documents in Elasticsearch

Elastic Stack Elasticsearch
1

Hi,

I'm collecting logs from an application recording the start time. Then another record is signifying the end time. Both records have to match on DeviceId and a CorrelationId. I'm using an ingest pipeline to process the logs. When the log line is processed that is the end time, I want to query ES and get the starttime and calculate the difference, and add a field for it.
I'm at a loss on how and if this can be done in pipelines in Elasticsearch.
I saw something similar via Logstash, but I'm trying to avoid Logstash and just do this in Elasticsearch.

Any ideas?
Thanks,
Tim

1 Like
2

Use an entity centric index - see https://youtu.be/yBf7oeJKH2Y

The comments include a link to example data and scripts

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

4

No they don't because I expect they are awaiting a review which is why only I can see them.

I've just updated the scripts to work with elasticsearch 6.3 and this is the link