Problems with Logstash Parsing

Elastic Stack Logstash
1

Hi Guys,
My first post so please bear with me. I have the following config in Logstash:

input {
file {
path => "/var/log/messages"
start_position => "beginning"
}
}

filter {

grok {
match => { "message" => "%{CISCOTIMESTAMP: Time_Logged} %{IP:Host_IP} 20%{DATESTAMP:Time_Logged} %{IP:Source} 20%{DATESTAMP:Event_Time} (?<Hostname>[^,]*) %{DATA:Misc} %{GREEDYDATA:Message_Text}"
}

if "_grokparsefailure" in [tags] {
drop { }
}

}
}

output {
elasticsearch {
hosts => ["10.0.0.21:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
}
stdout {
codec => rubydebug
}
}

This was all working fine couple of days ago and all I did with comment out the _grokparsefailure part. Now when I try to uncomment it, it's not working. Am I missing any brackets etc? I get the following error below:

[root@server conf.d]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f logstash.conf

Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties [2018-12-03T11:43:04,481][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2018-12-03T11:43:04,508][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.5.1"} [2018-12-03T11:43:06,600][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 27, column 4 (byte 441) after filter {\n\n grok {\n match => { \"message\" => \"%{CISCOTIMESTAMP: Time_Logged} %{IP:Host_IP} 20%{DATESTAMP:Time_Logged} %{IP:Source} 20%{DATESTAMP:Event_Time} (?<Hostname>[^,]*) %{DATA:Misc} %{GREEDYDATA:Message_Text}\"\n }\n\nif ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:incompile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:inblock in compile_sources'", "org/jruby/RubyArray.java:2486:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:incompile_sources'", "org/logstash/execution/AbstractPipelineExt.java:149:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:22:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:42:inblock in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:92:in block in exclusive'", "org/jruby/ext/thread/Mutex.java:148:insynchronize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:92:in exclusive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:38:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:317:in block in converge_state'"]} [2018-12-03T11:43:07,003][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties [2018-12-03T11:43:04,481][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2018-12-03T11:43:04,508][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.5.1"} [2018-12-03T11:43:06,600][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 27, column 4 (byte 441) after filter {\n\n grok {\n match => { \"message\" => \"%{CISCOTIMESTAMP: Time_Logged} %{IP:Host_IP} 20%{DATESTAMP:Time_Logged} %{IP:Source} 20%{DATESTAMP:Event_Time} (?<Hostname>[^,]*) %{DATA:Misc} %{GREEDYDATA:Message_Text}\"\n }\n\nif ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:incompile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:inblock in compile_sources'", "org/jruby/RubyArray.java:2486:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:incompile_sources'", "org/logstash/execution/AbstractPipelineExt.java:149:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:22:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:42:inblock in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:92:in block in exclusive'", "org/jruby/ext/thread/Mutex.java:148:insynchronize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:92:in exclusive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:38:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:317:in block in converge_state'"]} [2018-12-03T11:43:07,003][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}[root@aasvlgt01 conf.d]# /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f logstash.conf

2

Hi Guys,
Just a quick update. It started working. It was an issue with the sequence of the brackets. I only had to move the if statement below one bracket and it worked. This is what it looks like now:

filter {

grok {
match => { "message" => "%{CISCOTIMESTAMP: Time_Logged} %{IP:Host_IP} 20%{DATESTAMP:Time_Logged} %{IP:Source} 20%{DATESTAMP:Event_Time} (?<Hostname>[^,]*) %{DATA:Misc} %{GREEDYDATA:Message_Text}"
}
}
if "_grokparsefailure" in [tags] {
drop { }
}

}

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.