Merging information from two log files

Elastic Stack Logstash
1

Hi ,

I need to merge the information from two log file
ex: From the first log file I am getting some log information and second log file I am getting ID , I want to append that ID with each logs coming from first file..

I am doing it like

input
{
file
{
path => "D:/log_file/insightlogfolder/*"
sincedb_path => "D:/data1.sincedb"
type => "logfile"
start_position => "beginning"
}

	file 
	{
		path => "D:/log_file/ID/*"
		type => "pidlog"
		sincedb_path => "D:/data2.sincedb"
		start_position => "beginning"
	}	
}
filter       
{
	grok 
	{
		match => [ "message", "%{NUMBER:id}" ]

	}
	if [type] == "logfile"
	{
		grok 
		{
			patterns_dir => "D:/log_file/patterns"
			match => [ "message", "%{MASTER_LOG}" ]
			add_field => { "errorlog" => "errorlog" }
		}
	}
	mutate 
	{
			add_field => { "ID" => "%{id}" }
	}

}

But not getting the desired result,
please anyone can help

2

Logstash doesn't support that kind of log merging. Not with the stock plugins anyway. It would certainly be possible to write a custom plugin for it.

1 Like
3

@magnusbaeck I was going through some posts , I came accross plugin "logstash-filter-memorize" , can it be done through this ?

4

Possibly, yes, but I'd be worried about corner cases. It definitely looks like it would work if both records that you want to merge come from the same file so that their internal order is fixed. But if they come from different files, how can you be sure that the records are read in the order required?

5

Hi,

Is there any update on this thread ? Will it be possible by any chance to get information from two log files merged?