How do i aggregate these documets

Elastic Stack Logstash
1

I have logs of around thousand lines in a file, from which only three lines are useful for me. I am using grok filter to identify and filter those logs, now i want them to be in same document. How do i do that?
I am running it for similar multiple files. I find path is only common field for all three documents of a single log file.
`
{
"message" => "[INFO main : 10/08/2019-23:49:43.286 : EDeliveryProcessor] COBDate: 10/08/2019",
"@version" => "1",
"SLA_Start" => 2019-12-05T19:00:00.000Z,
"path" => "/opt/logstash/testlogs/runOfflinesC-DailyStmt.log",
"COBDate" => 2019-01-07T18:30:00.000Z,
"@timestamp" => 2019-12-06T10:35:58.510Z,
"host" => "sd-0153-fc25.nam.nsroot.net",
"sys" => "[INFO main : 10/08/2019-23:49:43.286 : EDeliveryProcessor]",
"SLA_End" => 2019-12-05T19:30:00.000Z
}

{
"message" => " Offline batch Start Time ->Oct 8, 2019 11:47:33 PM, End time ->Oct 8, 2019 11:49:43 PM",
"@version" => "1",
"Start Time" => 2019-10-08T18:17:33.000Z,
"path" => "/opt/logstash/testlogs/runOfflinesC-DailyStmt.log",
"SLA_Start" => 2019-12-05T19:00:00.000Z,
"End Time" => 2019-10-08T18:19:43.000Z,
"@timestamp" => 2019-12-06T10:35:58.509Z,
"host" => "sd-0153-fc25.nam.nsroot.net",
"SLA_End" => 2019-12-05T19:30:00.000Z
}

{
"message" => "[INFO main : 10/08/2019-23:49:43.286 : EDeliveryProcessor] Production-C:DAR:DailyStmt: EDeliveryDriver2 (10/08/2019) : Processed <176> Users <0> Jobs Failed",
"@version" => "1",
"SLA_Start" => 2019-12-05T19:00:00.000Z,
"path" => "/opt/logstash/testlogs/runOfflinesC-DailyStmt.log",
"date" => "10/08/2019",
"@timestamp" => 2019-12-06T10:35:58.510Z,
"host" => "sd-0153-fc25.nam.nsroot.net",
"sys" => "[INFO main : 10/08/2019-23:49:43.286 : EDeliveryProcessor]",
"Processed User" => 176,
"Failed User" => 0,
"SLA_End" => 2019-12-05T19:30:00.000Z
}

2

You might be able to use an aggregate filter based on [path].

1 Like
3

I need to give a code setting i am missing, can you help me writing aggregate filter part.
Currently I'm using below code -
aggregate{
task_id => "%{path}"
}
As i have one field that is consistent and common ie Path, but there are many fields in document which are overlapping.

4

What am i missing in this?

 if "" in ['Start Time']{
	aggregate{
		task_id => "%{path}"
		code => "
			map['Start Time'] = event.get('Start Time');
			map['End Time'] = event.get('End Time');
			event.cancel()
		"
		map_action => create
	}
	}

if "" in ['Processed User']{
aggregate{
	task_id => "%{path}"
	code => "
		map['Processed User'] = event.get('Processed User');
		map['Failed User'] = event.get('Failed User');
		event.cancel()
	"
	#map_action => update
}
}

if "" in ['COBDate']{
aggregate{
	task_id => "%{path}"
	code => "
		map['COBDate'] = event.get('COBDate');
		map['SLA_Start'] = event.get('SLA_Start');
		map['SLA_End'] = event.get('SLA_End');
		event.cancel()
		#map_meta.timeout = 0
	"
	#map_action => update
	push_previous_map_as_event => true
	end_of_task => true
	
	#timeout
}
}
5

What are you trying to test using that?

8

I was using it wrong, if [Start Time ] worked for me.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.