How to calculate elapsedtime using logstash

[klymptzic]

Hi Everyone,

I need to extract response time of an event by calculating elapsedtime from start and end time

excerpts from log:
08-12-17 04:38:17:090 [INFO] [thread-1] [User1] c.s.s.i.a.t.JavaClass1 - Loading Data from
08-12-17 04:49:38:902 [INFO] [thread-10] [User1] c.s.s.i.a.t.JavaClass2 - Data loaded: #12

I need calc the elapsed time between 08-12-17 04:38:17 & 08-12-17 04:49:38:902. The difference is the time taken to load data that I need to plot/visualize in Kibana.

Need to plot this for multiple users who had performed the event
User1 - 0:11:21
User2 - 0:10:01
for instance..

Appreciate your help.

[guyboertje]

Can you do this as a calculated value in Kibana?

If not try the elapsed filter https://www.elastic.co/guide/en/logstash/6.0/plugins-filters-elapsed.html

With this filter though, you should understand that it needs to "see" all events in original order. Due to the fact that LS runs in a multi-threaded way there is no guarantee that a thread that takes a batch with a "start" event will have stored the start event in the elapsed filter before the second thread processes the "stop" event. Say Batch 1 has 125 events and the start event is number 121 and in Batch 2 the stop event is number 5 - it follows that thread 1 (with batch 1) needs to process 120 events before it gets to the start event whereas thread 2 (with batch 2) only needs to process 4 events before it reaches the stop event.
Therefore you need to set pipeline.workers: 1 in your logstash.yml. This is why I suggested the Kibana method if you can get it to work.