Logstash Aggregate with multiple end tags

[Nani_20]

Hi All,

I am trying to aggregate elapsed time value for a specific event. But, the event end tags are either one of the two tags (either Saved or Modified) as specified below in the sample data.

Sample Data with columns ID, Event_tag

1 Start
1 Saved
2 Start
2 Saved
2 Modified
3 Start
3 Saved
3 Modified
4 Start
4 Saved

I would like to find the time taken by a unique ID i.e. elapsed time between Start & End (either Saved or Modified).
I tried with Elapsed, but it requires a single end tag. Aggregate can do this by adding a new event using push_map_as_event_on_timeout. But, I would like to store it in the end row of each event. Is it possible?

Any help is appreciated. Thanks in advance.

[ptamba]

if you are sure that the event tag only have either Saved or Saved, Modified , then you can access the modified timestamp with %{event_tag[1]}. since they are on the same column, i imagine the eveng_tag will be in array.

then you could go with :

if “Modified” in [event_tag] { 
  filter { 
    #elapsed filter with %{event_tag[1]} as end value 
  } 
} 

else { 
  filter {
     #elapsed filter with %{event_tag[0]} as end value
  } 
} 

the best way will be using ruby filter to extract the time stamp from either Saved or Modified. Here’s an example

[Nani_20]

@ptamba I tried the above solution. I was getting elapsed_end_without_start exception.
I tried Ruby filter earlier, I was able to capture times for Saved & Modified. But, I am not sure how can I ignore Saved when there is a Modified event for the given ID.

[Nani_20]

@ptamba Event tag is not an array. It's a field with values - Start, Saved, Modified. For a given ID, all three or only Start and Saved may occur.
Thank You for your response. Is there is any way to achieve the above requirement?
TIA

[ptamba]

can you show a sample event, and the current output from logstash stdout ?

if i understand correctly, and event may have event_tag.start and event_tag.saved or event_tag.start with event_tag.saved and event_tag.modified ? and you want to add a new field called elapsed time?

[Nani_20]

@ptamba Sorry for the confusion, I hope the below sample log file and current output will make things clear.

Sample Log File

       TIMESTAMP   			EVENT	 ID
Apr 28, 2020 @ 15:17:22.337 Start	 1
Apr 28, 2020 @ 15:17:23.215 Saved	 1
Apr 28, 2020 @ 15:17:24.440 Start	 2
Apr 28, 2020 @ 15:17:24.964 Saved	 2
Apr 28, 2020 @ 15:17:25.359 Modified 2
Apr 28, 2020 @ 16:18:29.587 Start	 3
Apr 28, 2020 @ 16:18:31.562 Saved	 3
Apr 28, 2020 @ 16:18:31.914 Modified 3
Apr 28, 2020 @ 20:07:52.946 Start	 4
Apr 28, 2020 @ 20:07:53.304 Saved	 4

Current output using Ruby filter

Apr 28, 2020 @ 15:17:22.337 Start		- 		1
Apr 28, 2020 @ 15:17:23.215 Saved		0.878	1
Apr 28, 2020 @ 15:17:24.440 Start		- 		2
Apr 28, 2020 @ 15:17:24.964 Saved		0.524	2
Apr 28, 2020 @ 15:17:25.359 Modified	0.919	2
Apr 28, 2020 @ 16:18:29.587 Start	 	-		3
Apr 28, 2020 @ 16:18:31.562 Saved		1.975	3
Apr 28, 2020 @ 16:18:31.914 Modified	2.327	3
Apr 28, 2020 @ 20:07:52.946 Start	 	-		4
Apr 28, 2020 @ 20:07:53.304 Saved		0.358	4

But, I would like to delete the Saved time if the Modified event exists for a given ID. Find below the desired output for above example.

Desired output

Apr 28, 2020 @ 15:17:22.337 Start		- 		1
Apr 28, 2020 @ 15:17:23.215 Saved		0.878	1
Apr 28, 2020 @ 15:17:24.440 Start		- 		2
Apr 28, 2020 @ 15:17:24.964 Saved		-       2
Apr 28, 2020 @ 15:17:25.359 Modified	0.919	2
Apr 28, 2020 @ 16:18:29.587 Start	 	-		3
Apr 28, 2020 @ 16:18:31.562 Saved		-    	3
Apr 28, 2020 @ 16:18:31.914 Modified	2.327	3
Apr 28, 2020 @ 20:07:52.946 Start	 	-		4
Apr 28, 2020 @ 20:07:53.304 Saved		0.358	4