Calculate time difference between 2 lines in log file if there are multiple sets

Elastic Stack Logstash
1

Hi everyone,

I retrieved a log file that look like this :

2019-03-07 04:35:57.421     19EC | INFO    TIMEDATA.DLL:         Capacity allocated: Hashtable entries 20648881, Heap mem 805306368
2019-03-07 04:35:57.421     19EC | INFO    DPREAD:               Data import started: 07.03.19 04:35:57
2019-03-07 04:35:57.421     19EC | INFO    DPREAD:               Importing file: C:\path\to\file1
2019-03-07 04:35:57.452     19EC | INFO    DPREAD:               Completed importing file: C:\path\to\file1
2019-03-07 04:35:57.452     19EC | INFO    DPREAD:               Importing file: C:\path\to\file2
2019-03-07 04:36:43.545     19EC | INFO    DPREAD:               Completed importing file: C:\path\to\file2
2019-03-07 05:38:55.332     19EC | INFO    TMDPDATA-INIT:        Datasupply info: vwdpm.dcsDefault.2.0
2019-03-07 06:40:50.421     19EC | INFO    DPREAD:               Importing file: C:\path\to\file1
2019-03-07 06:40:55.452     19EC | INFO    DPREAD:               Completed importing file: C:\path\to\file1

What i want to do is to calculate the difference between the time of "Importing file: C:\path\to\file1" and "Completed importing file: C:\path\to\file1".

There are 2 sets matching this criteria in the log file and I want Elapsed time for both sets.
I used Logstash filter (aggregate and elapsed) but able to retrieve time only for 1 set.
Could someone help me on this?

2

What have you tried and what do you not like about the results?

3

Actually, when i run my code, it behaves very weird.
sometimes it gives elapsed_time for both events, sometimes it gives only for one event and sometimes it pairs file1 with file2.

It seems continuity or sequence of lines is not followed by code.

Hence, I made a change in the settings of logstash.yml file.
I set "pipeline.workers: 1" so that it does not do parallel processing.

But, it will impact my performance.
Any work around for this?

4

You need both pipeline.workers 1 and setting java_execution false if you are running version 7.

Yes, this limits your scalability. No, there is no workaround.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.