Calculate time difference between two logs with uniqueid

Elastic Stack Logstash
1

hi there

is there anyway to calculate time difference between two logs with uniqueid?

eg i create fields log_timestamp, log_type, user_id, mobile_no, etc., below logs

2018-09-19 09:24:57,113 (null) 21 WARN : Test :: vh1o4ctmdpn3aft53o5nx0nh :: User Req : <UserRequest><version>9.7</version><userID>458458000000018</userID><userCode>2C10000299</userCode><desc>123456</desc><desc>123456</desc><mobileNo></mobileNo></UserRequest>

2018-09-19 09:24:57,453 (null) 21 WARN : Test :: vh1o4ctmdpn3aft53o5nx0nh :: User Res : <UserResponse><version>9.7</version><userID>458458000000018</userID><userCode>2C10000299</userCode><desc>123456</desc><desc>123456</desc><mobileNo></mobileNo></UserResponse>

i am in the need to calculate log_timestamp difference between these two logs using user_id.
how to link user_id for them? my logstash pipeline is eating a lot of logs at the same time.

2

Use the "elapsed" plugin and set your worker threads to 1 so that multiple threads don't miss the end_events from the elapsed plugin.
Elapsed Plugin - Logstash Docs

3

hi @AquaX

thanks very much for advice.
but i'm being used 2 threads long time ago
any other options??

4

How long is “long”? You can adjust the timeout setting on the elapsed filter, it will require more events to be held in memory for longer but as long as you don’t restart your instance that should be ok.

I can’t think of any other alternative at the moment unless you do some crazy setup like scanning your log for your starting event, save that to elastic and if you find an ending event save that to a new log file. The new log file will then have its own log stash config that will query elastic for the starting event and then you may be able to do something with it. This is just theory and it would have to be tested... not sure that it will be possible.

5

thank you for your adivise @AquaX
but my logs are consecutive streaming.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.