Considering the last selection log

[satyajeet_dutt]

hi there,
I am stuck with a specific issue related to clickstream log. It's based on the selection made by the user.If a user selects something then the request gets added up in the request parameter. I am posting a sample log over here so that you can better understand the problem

->2019-03-28T09:41:15,273 hostname pid module thread someid POST selectionguides/searchbyid/DEMO/test2/FRA/FR {languageID=FR, selectionGuideGlobalID=test2, projectId=DEMO, countryID=FRA} {} {"A":"X"} 200 152
->2019-03-28T09:41:22,989 hostname pid module thread someid POST selectionguides/searchbyid/DEMO/test2/FRA/FR {languageID=FR, selectionGuideGlobalID=test2, projectId=DEMO, countryID=FRA} {} {"A":"X","B":"Y"} 200 138
->2019-03-28T09:41:23,900 hostname pid module thread someid POST selectionguides/searchbyid/DEMO/test2/FRA/FR {languageID=FR, selectionGuideGlobalID=test2, projectId=DEMO, countryID=FRA} {} {"A":"X","B":"Y","C":"Z"} 200 88

In the above log, you can see that the request body which is at the last gets added up everytime with the selection made by user like this:-

{"A":"X"}
{"A":"X","B":"Y"}
{"A":"X","B":"Y","C":"Z"}

I want the last log only for my purpose.So, please recommend me something for this.

[satyajeet_dutt]

Write now I am trying with aggregation filter in logstash by putting if-else condition.

[Badger]

I would suggest something like example 3 in the aggregate documentation, but keep overwriting the map entry with the most recent value of the request body.

[satyajeet_dutt]

Is there any way to take the last log with the latest timestamp??In this way, only the last log will be available for my analysis.

Then I'll aggregate based on my requirement

[satyajeet_dutt]

hi there,
Here I have created one list 'request_body'.
map['request_body'] << {'request' => event.get('request')}
Can u tell me that how should I access the last element of that list. I just want the syntax

[Badger]

If you only want the last entry then do not append using <<, just overwrite using =

[satyajeet_dutt]

I tried that, but now it is giving output for every request. Basically my aggregation is not working .I wanted it first to be aggregated based on user id and stored in one list('request_body') by using this-> map['request_body'] << {'request' => event.get('request')}
Then I wanted to select the last element from that list.

[satyajeet_dutt]

I have written this code for aggreagation:->

            aggregate {
				task_id => "%{[CorrelationId][paths]}"
				code => "
				map['CorrelationId'] = event.get('CorrelationId')
				map['paths'] = event.get('paths')
				map['request_body'] ||= []
				if (event.get('request') != nil)
				 if !( map['request_body'].include? event.get('request') )
				  map['request_body'] =  event.get('request')
				 
                end
				end 
				event.cancel()
				"
				push_previous_map_as_event => true
				timeout => 3
	

   		  }

I don't know why it is giving me output for each and every logs

[satyajeet_dutt]

I am getting my request_body as
For 1st log
request_body{
"A=X"
}
For 2nd log
request_body{
"A"="X",
"B"="Y",
}
but I want the last request_body only.
plz help

[Badger]

You have not shown us what an event looks like, so we cannot say what is wrong with the aggregate filter. Either show us the filter {} section, or show us an event with output { stdout { codec => rubydebug } }

[satyajeet_dutt]

Check the request_body part, for one log I am getting
request_body{
"A"="X"
}
for another log I am getting
request_body{
"A"="X",
"B"="Y",
}
So, basically I want the final one(2nd one),as it contains the element of the previous log also.

[satyajeet_dutt]

Please reply