How to put one message to several indices by condition?

Hi there.
I have a bunch of log messages that I want to parse by several ways.
All messages are guaranteed to fit one common grok filter so they are going to index 1.
But i want to check this messages agains other grok filters and depending of results send the same messages to other indices too.
I wrote processing config below but now I'm confused how to make output config.
As I understand after processing it's still one message out but I need at least two messages for two different ES indices.

if [log_type] == "common_type"
{
	if "condition1" in [message]
	{
		grok
		{
			match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp}%{SPACE}%{NOTSPACE}%{SPACE}%{WORD:thread}%{NOTSPACE}%{SPACE}%{NOTSPACE:username}%{SPACE}%{NOTSPACE:log_level}%{SPACE}%{NOTSPACE:main}%{SPACE}%{GREEDYDATA:_log}"]
		}
		mutate
		{
			add_field => { "log_subtype" => "common_type-sl" }				
		}
	}
	
	else if "condition2" in [message]
	{
		grok
		{
			match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp}%{SPACE}%{NOTSPACE}%{SPACE}%{WORD:thread}%{NOTSPACE}%{SPACE}%{NOTSPACE:username}%{SPACE}%{NOTSPACE:log_level}%{SPACE}%{NOTSPACE:main}%{SPACE}%{GREEDYDATA:_log}"]
		}			
		mutate
		{
			add_field => { "log_subtype" => "common_type-ir" }				
		}
	}
	##common filter here
	grok
	{
		match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp}%{SPACE}%{NOTSPACE}%{SPACE}%{WORD:thread}%{NOTSPACE}%{SPACE}%{NOTSPACE:username}%{SPACE}%{NOTSPACE:log_level}%{SPACE}%{NOTSPACE:main}%{SPACE}%{GREEDYDATA:_log}"]
	}
	date
	{
		match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss,SSS"]
		timezone => "UTC"
		add_tag => [ "tsmatch" ]
	}
	mutate
	{
		remove_field => ["log_timestamp", "@message", "message","source","offset" ]
	}
}

You can use conditionals in the output section just as you can in the filter section of the configuration.

But how to make sure that one message will be processed exaxtly two times by different grok filter?
If I understand right, when message come to output it will be parsed by latest applied filter, won't it?

If you need two copies of the event to go through differernt sets of filters before being output then write them to two different pipelines.

I'm not sure that i got you right.
I though that no matter how many grok filters i apply. At the end I'll get one set of fields but i need two sets of different fields from one message.

Look at pipeline-to-pipeline communication. In particular the forked path pattern.

Thank you. That is exactly what I need.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.