How to put one message to several indices by condition?

Elastic Stack Logstash
#1

Hi there.
I have a bunch of log messages that I want to parse by several ways.
All messages are guaranteed to fit one common grok filter so they are going to index 1.
But i want to check this messages agains other grok filters and depending of results send the same messages to other indices too.
I wrote processing config below but now I'm confused how to make output config.
As I understand after processing it's still one message out but I need at least two messages for two different ES indices.

if [log_type] == "common_type"
{
	if "condition1" in [message]
	{
		grok
		{
			match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp}%{SPACE}%{NOTSPACE}%{SPACE}%{WORD:thread}%{NOTSPACE}%{SPACE}%{NOTSPACE:username}%{SPACE}%{NOTSPACE:log_level}%{SPACE}%{NOTSPACE:main}%{SPACE}%{GREEDYDATA:_log}"]
		}
		mutate
		{
			add_field => { "log_subtype" => "common_type-sl" }				
		}
	}
	
	else if "condition2" in [message]
	{
		grok
		{
			match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp}%{SPACE}%{NOTSPACE}%{SPACE}%{WORD:thread}%{NOTSPACE}%{SPACE}%{NOTSPACE:username}%{SPACE}%{NOTSPACE:log_level}%{SPACE}%{NOTSPACE:main}%{SPACE}%{GREEDYDATA:_log}"]
		}			
		mutate
		{
			add_field => { "log_subtype" => "common_type-ir" }				
		}
	}
	##common filter here
	grok
	{
		match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp}%{SPACE}%{NOTSPACE}%{SPACE}%{WORD:thread}%{NOTSPACE}%{SPACE}%{NOTSPACE:username}%{SPACE}%{NOTSPACE:log_level}%{SPACE}%{NOTSPACE:main}%{SPACE}%{GREEDYDATA:_log}"]
	}
	date
	{
		match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss,SSS"]
		timezone => "UTC"
		add_tag => [ "tsmatch" ]
	}
	mutate
	{
		remove_field => ["log_timestamp", "@message", "message","source","offset" ]
	}
}
#2

You can use conditionals in the output section just as you can in the filter section of the configuration.

#3

But how to make sure that one message will be processed exaxtly two times by different grok filter?
If I understand right, when message come to output it will be parsed by latest applied filter, won't it?

#4

If you need two copies of the event to go through differernt sets of filters before being output then write them to two different pipelines.

#5

I'm not sure that i got you right.
I though that no matter how many grok filters i apply. At the end I'll get one set of fields but i need two sets of different fields from one message.

#6

Look at pipeline-to-pipeline communication. In particular the forked path pattern.

#8

Thank you. That is exactly what I need.

#9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.