Add a time field in elasticsearch and calculate time between two event

[The-morpho]

I'm new to Elasticsearch.

I’m working with log files stored in Elasticsearch, that contains information about some operations on differents levels, I have a field timestamp that represents the time when each operation has occured on each step.

This is my work flow:

-folder-> filebeat -> logstash -> elasticsearch <- kibana

I want to calculate the time between the start and the end of the process. between step1 and step 3 and add it to a field that I will create so I can visualize it on Kibana.

This is an exemple of some of my logs.

{“log_level”:“INFO”,“timestamp”:“2021-12-22T11:49:06.124890Z”,“event_type”:“step1”,“mid”:“96712abc”}{“log_level”:“INFO”,“timestamp”:“2021-12-22T11:49:07.124890Z”,“event_type”:“step2”,“mid”:“96712abc”} {“log_level”:“INFO”,“timestamp”:“2021-12-22T11:49:08.124890Z”,“event_type”:“step3”,“mid”:“96712abc”}

[Badger]

Use an aggregate filter. You may want something like example 1 or example 3 depending on whether the steps are fixed or variable.

[The-morpho]

Thank your for your reply, In the first exemple:

match => [ "message", "%{LOGLEVEL:loglevel} - %{NOTSPACE:taskid} - %{NOTSPACE:logger} - %{WORD:label}( - %{INT:duration:int})?"

What is the role of the question mark ?

[Badger]

@The-morpho

What is the role of the question mark ?

? means "zero or more", so the ( - %{INT:duration:int}) part is optional. It occurs on the SQL lines, but not the start and end.