Add a time field in elasticsearch and calculate time between two event

I'm new to Elasticsearch.

I’m working with log files stored in Elasticsearch, that contains information about some operations on differents levels, I have a field timestamp that represents the time when each operation has occured on each step.

This is my work flow:

-folder-> filebeat -> logstash -> Elasticsearch <- kibana

I want to calculate the time between the start and the end of the process. between step1 and step 3 and add it to a field that I will create so I can visualize it on Kibana.

This is an exemple of some of my logs.

{“log_level”:“INFO”,“timestamp”:“2021-12-22T11:49:06.124890Z”,“event_type”:“step1”,“mid”:“96712abc”}{“log_level”:“INFO”,“timestamp”:“2021-12-22T11:49:07.124890Z”,“event_type”:“step2”,“mid”:“96712abc”} {“log_level”:“INFO”,“timestamp”:“2021-12-22T11:49:08.124890Z”,“event_type”:“step3”,“mid”:“96712abc”}

Use an aggregate filter. You may want something like example 1 or example 3 depending on whether the steps are fixed or variable.

1 Like

Thank your for your reply, In the first exemple:

match => [ "message", "%{LOGLEVEL:loglevel} - %{NOTSPACE:taskid} - %{NOTSPACE:logger} - %{WORD:label}( - %{INT:duration:int})?"

What is the role of the question mark ?

? means "zero or more", so the ( - %{INT:duration:int}) part is optional. It occurs on the SQL lines, but not the start and end.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.