File beat --> logtash --> conditional output

Elastic Stack Logstash
1

Hi

I did research in this forum and documentation but couldn't find any solution so posting here for help.

OS : windows 10
Filebeat : 6.4.2 sending 2 CSV files to Logstash
File name London.csv , Australia.csv

Logstash config has following for output.

output {
elasticsearch {
if ([source] =~ \London.csv)
{
hosts => "http://myESSErver:port"
manage_template => true
index => "London-%{+YYYY.MM.dd}"

 }

if ([source] =~ \Australia.csv)
{
hosts => "http://myESSErver:port"
manage_template => true
index => "Australia-%{+YYYY.MM.dd}"

 }

}

When i start my logstash I get error like

[2018-11-07T13:56:16,779][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 15, column 7 (byte 2230) after output {\n elasticsearch {\n if "

So am i doing something which i m not supposed to ?

2

You need a closing backslash for all regular expressions.

3

so i have tried

if ([source] =~ "\London.csv\")

if ([source] =~ \London.csv\)

and both time it failed.

4

Can you show us what the event and the source field look like?

5

sure give me 5 mins.

6

Not diverting topic but just wanted to share this with you first.

So if i try following

input {
beats {
port => "5045"
}
}
filter {
csv {
separator => ","
columns => ["Filed1" , "Filed2" , "Filed3"]
}

}
output {
elasticsearch {

        hosts => "http://myesserver:port"
        manage_template => true
        index => "London-%{+YYYY.MM.dd}"

}
stdout {}
}

my logstash agent starts up fine.

So i stop the logstash and add conditional statement

input {
beats {
port => "5045"
}
}
filter {
csv {
separator => ","
columns => ["Filed1" , "Filed2" , "Filed3"]
}

if ([source] =~ \London.csv\)
{
add_field => [ "Location", "London" ]
}

}
output {
elasticsearch {

        hosts => "http://myesserver:port"
        manage_template => true
        index => "London-%{+YYYY.MM.dd}"

}
stdout {}
}

Error

[2018-11-07T15:53:13,144][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, ", ', / at line 12, column 19 (byte 167) after filter {\n csv {\n separator => ","\n columns => ["Filed1" , "Filed2" , "Filed3"]\n }\n \n if ([source] =~ ",

7

another update.

first my IF had wrong syntax i think.

so when do this

input {
beats {
port => "5045"
}
}
filter {
csv {
separator => ","
columns => ["Filed1" , "Filed2" , "Filed3"]
}

if [source] == "C:\mycsv\London.csv"
{

   }

}
output {
elasticsearch {

        hosts => "http://127.0.0.1:9200"
        manage_template => true
        index => "London-%{+YYYY.MM.dd}"

}
stdout {}
}

My logstash starts u fine.

But if i add this

if [source] == "C:\mycsv\London.csv"
{
add_field => [ "Location", "London" ]
}

it fails.

[2018-11-07T16:05:39,470][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 14, column 24 (byte 271) after filter {\n csv {\n separator => ","\n columns => ["Filed1" , "Filed2" , "Filed3"]\n }\n \n if [source] == "C:\mycsv\London.csv"\n {\n add_field ",

8

Sorry for posting too many updates :slight_smile:

add_field needs to be in mutate so my issue is fixed. Will update if ingestion of data works without any issues.

thanks for all the help

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.