How to calculate time difference between 2 log lines for a unique ID

Elastic Stack Kibana
1

I need to calculate the difference in time between 2 log lines for http code 200. The difference should be done for a unique id. Please find the log line below.

2018/01/16 00:13:44.890 [HCServiceImpl] [**qtp5720769-243**]:  AUDIT- INFO:  Request received : /mdp/content?sourceType=cid&pset=mdp%3Ano-presentation&filter%3AcontentId=cid%3A%2F%2Fprogramid%253A%252F%252F2090543776%23programid%253A%252F%252F1264605229

2018/01/16 00:13:44.897 [HCServiceImpl] [**qtp5720769-243**]:  AUDIT- INFO:  Returning response code : 200

Here qtp5720769-243 is the Unique ID. Then check for 200. Then calculate the time between [ 00:13:44.897 ] & [ 00:13:44.890 ]. I am not sure how to do this. Is this achievable using scripted field?

2

Elasticsearch doesn't support that kind of operation at query time, you'd have to do it another way. Or, maybe you could use pipeline aggs, but I don't really understand how those work enough to even tell you if that's something they can be used for. Kibana has some support for them, you could play around and see.

You could get that data at ingest time though by enriching it and adding a new field with that value. Basically, you check to see when the newest previous document with the same criteria, calculate the difference on the document you're about to index, and then index it with that calculation. Something like injest node (you can read more about that here) or log logstash should help.

If you need to do this at query time, you could figure out the difference using two queries and a little math, but Kibana doesn't support that.

3

Thanks for the reply. I will check for the information you provided.

4

I am trying to resolve the same problem using logstash using elasticsearch filter
see example
https://www.elastic.co/guide/en/logstash/current/lookup-enrichment.html
in principle during parsing of line it checks if type == end , query index to search for start
retrieve data and do calculation of duration

the problem of this approach:
if there are 2 lines close to each other, the query is executed too fast (before the first line is inserted into elastic index) so it does not retrieve the data.
It works only if the process is slowed down or if there is enough lines between these two lines to give logstash time to PUT the first line into index.

5

Thanks Petr

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.