# Calculate time difference between 2 lines in log file if there are multiple sets

**URL:** <https://discuss.elastic.co/t/calculate-time-difference-between-2-lines-in-log-file-if-there-are-multiple-sets/228856>\
**Category:** Logstash\
**Created:** [April 20, 2020, 11:58am UTC](https://discuss.elastic.co/t/calculate-time-difference-between-2-lines-in-log-file-if-there-are-multiple-sets/228856 "2020-04-20T11:58:31Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![airan](https://avatars.discourse-cdn.com/v4/letter/a/b38774/32.png) [@airan](https://discuss.elastic.co/u/airan)\
**Post date:** [April 20, 2020, 11:58am UTC](https://discuss.elastic.co/t/calculate-time-difference-between-2-lines-in-log-file-if-there-are-multiple-sets/228856/1 "2020-04-20T11:58:31Z")

</div>

Hi everyone,

I retrieved a log file that look like this :

```auto
2019-03-07 04:35:57.421 19EC | INFO TIMEDATA.DLL: Capacity allocated: Hashtable entries 20648881, Heap mem 805306368
2019-03-07 04:35:57.421 19EC | INFO DPREAD: Data import started: 07.03.19 04:35:57
2019-03-07 04:35:57.421 19EC | INFO DPREAD: Importing file: C:\path\to\file1
2019-03-07 04:35:57.452 19EC | INFO DPREAD: Completed importing file: C:\path\to\file1
2019-03-07 04:35:57.452 19EC | INFO DPREAD: Importing file: C:\path\to\file2
2019-03-07 04:36:43.545 19EC | INFO DPREAD: Completed importing file: C:\path\to\file2
2019-03-07 05:38:55.332 19EC | INFO TMDPDATA-INIT: Datasupply info: vwdpm.dcsDefault.2.0
2019-03-07 06:40:50.421 19EC | INFO DPREAD: Importing file: C:\path\to\file1
2019-03-07 06:40:55.452 19EC | INFO DPREAD: Completed importing file: C:\path\to\file1

```

What i want to do is to calculate the difference between the time of "Importing file: C:\path\to\file1" and "Completed importing file: C:\path\to\file1".

There are 2 sets matching this criteria in the log file and I want Elapsed time for both sets.  
I used Logstash filter (aggregate and elapsed) but able to retrieve time only for 1 set.  
Could someone help me on this?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 20, 2020, 3:11pm UTC](https://discuss.elastic.co/t/calculate-time-difference-between-2-lines-in-log-file-if-there-are-multiple-sets/228856/2 "2020-04-20T15:11:24Z")

</div>

> [@airan](#):
>
> I used Logstash filter (aggregate and elapsed)

What have you tried and what do you not like about the results?

---

<div class="post-metadata">

**Author:** ![airan](https://avatars.discourse-cdn.com/v4/letter/a/b38774/32.png) [@airan](https://discuss.elastic.co/u/airan)\
**Post date:** [April 22, 2020, 4:25pm UTC](https://discuss.elastic.co/t/calculate-time-difference-between-2-lines-in-log-file-if-there-are-multiple-sets/228856/3 "2020-04-22T16:25:28Z")

</div>

Actually, when i run my code, it behaves very weird.  
sometimes it gives elapsed\_time for both events, sometimes it gives only for one event and sometimes it pairs file1 with file2.

It seems continuity or sequence of lines is not followed by code.

Hence, I made a change in the settings of logstash.yml file.  
I set "pipeline.workers: 1" so that it does not do parallel processing.

But, it will impact my performance.  
Any work around for this?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 22, 2020, 4:38pm UTC](https://discuss.elastic.co/t/calculate-time-difference-between-2-lines-in-log-file-if-there-are-multiple-sets/228856/4 "2020-04-22T16:38:56Z")

</div>

> [@airan](#):
>
> It seems continuity or sequence of lines is not followed by code.

You need both pipeline.workers 1 and setting [java\_execution false](https://github.com/elastic/logstash/issues/10938) if you are running version 7.

Yes, this limits your scalability. No, there is no workaround.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 20, 2020, 4:39pm UTC](https://discuss.elastic.co/t/calculate-time-difference-between-2-lines-in-log-file-if-there-are-multiple-sets/228856/5 "2020-05-20T16:39:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.