# Time between two timestamps in different files

**URL:** <https://discuss.elastic.co/t/time-between-two-timestamps-in-different-files/248327>\
**Category:** Elasticsearch\
**Created:** [September 11, 2020, 1:10pm UTC](https://discuss.elastic.co/t/time-between-two-timestamps-in-different-files/248327 "2020-09-11T13:10:59Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tommie](https://avatars.discourse-cdn.com/v4/letter/t/f0a364/32.png) [@Tommie](https://discuss.elastic.co/u/Tommie)\
**Post date:** [September 11, 2020, 1:10pm UTC](https://discuss.elastic.co/t/time-between-two-timestamps-in-different-files/248327/1 "2020-09-11T13:10:59Z")

</div>

Hi,  
Can you please advise me setting up the following?

There are multiple log files (csv) coming from multiple sources.  
The logfiles contains data from transactions which are first processed in system A, then processed in system B and so on.  
In all systems there's the same unique identifier per transaction.  
How can I calculate the difference between the end time given in the logfile from system A and the starttime of system B?  
I've tried using Elastisearch filter in Logstash. That approach fails when logfile B is processed before logfile A.

Example (unique ID = 12)

LOGFILE A  
ID timeA actionA  
12 00:00:00 start processing  
12 00:00:10 end processing \< 'elapsed' gives me 10 seconds

LOGFILE B  
ID timeB actionB  
12 00:00:11 start processing  
12 00:00:15 end processing \< 'elapsed' gives me 4 seconds

With the plugin 'elapsed' I successfully calculate the time it took per system.

Is there a way I can calculate the time between 'end processing' system A (0:00:10) and the 'start processing' of system B (00:00:11) regardless of the order in which the files are processed?

Help or advice is greatly appreciated!

---

<div class="post-metadata">

**Author:** ![Peter\_Steenbergen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/peter_steenbergen/32/22888_2.png) [@Peter\_Steenbergen](https://discuss.elastic.co/u/Peter_Steenbergen)\
**Post date:** [September 11, 2020, 1:46pm UTC](https://discuss.elastic.co/t/time-between-two-timestamps-in-different-files/248327/2 "2020-09-11T13:46:12Z")

</div>

Have you seen the docs on Lookup Enrichment:  
[https://www.elastic.co/guide/en/logstash/current/lookup-enrichment.html#lookup-plugins](https://www.elastic.co/guide/en/logstash/current/lookup-enrichment.html#lookup-plugins)

The second code block maybe of good use to you.

---

<div class="post-metadata">

**Author:** ![Tommie](https://avatars.discourse-cdn.com/v4/letter/t/f0a364/32.png) [@Tommie](https://discuss.elastic.co/u/Tommie)\
**Post date:** [September 11, 2020, 4:22pm UTC](https://discuss.elastic.co/t/time-between-two-timestamps-in-different-files/248327/3 "2020-09-11T16:22:01Z")

</div>

> [@Tommie](#):
>
> I've tried using Elastisearch filter in Logstash. That approach fails when logfile B is processed before logfile A.

Hi Peter,  
Thanks for your quick reaction!  
I've tried that solution: 'I've tried using Elastisearch filter in Logstash. That approach fails when logfile B is processed before logfile A.'  
It only works if the files are delivered in the right order (which they often don't).

Any other suggestion?

---

<div class="post-metadata">

**Author:** ![Peter\_Steenbergen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/peter_steenbergen/32/22888_2.png) [@Peter\_Steenbergen](https://discuss.elastic.co/u/Peter_Steenbergen)\
**Post date:** [September 11, 2020, 7:27pm UTC](https://discuss.elastic.co/t/time-between-two-timestamps-in-different-files/248327/4 "2020-09-11T19:27:53Z")

</div>

I am thinking a bit out of the box now. How soon do you need that data of the calculation?  
What you can do is store the calculation in a new field, if it has no results during the lookup, you can you do an \_update\_by\_query later on for the empty/missing calculated fields. Elasticsearch now also has an enrich/lookup processor to leverage.

Additional: For missing values if you need a value present if do not have it at that time you can index null\_value with an average so you wont come blank.

---

<div class="post-metadata">

**Author:** ![Tommie](https://avatars.discourse-cdn.com/v4/letter/t/f0a364/32.png) [@Tommie](https://discuss.elastic.co/u/Tommie)\
**Post date:** [September 14, 2020, 9:28am UTC](https://discuss.elastic.co/t/time-between-two-timestamps-in-different-files/248327/5 "2020-09-14T09:28:42Z")

</div>

Thanks Peter, that sounds like a good approach since I don't need the calculation on the fly.  
I'm going to give it a try!

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [September 14, 2020, 11:09am UTC](https://discuss.elastic.co/t/time-between-two-timestamps-in-different-files/248327/6 "2020-09-14T11:09:10Z")

</div>

Check out the [transforms api](https://www.elastic.co/guide/en/elasticsearch/reference/master/transform-overview.html) as a framework for bringing related data together.  
It can use aggregations relating to a common ID to fuse related data and then store the results. These aggregations can also use custom scripts to derive data like a duration

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 12, 2020, 11:09am UTC](https://discuss.elastic.co/t/time-between-two-timestamps-in-different-files/248327/7 "2020-10-12T11:09:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.