Logstash grok vs. grok debugger

Elastic Stack Logstash
1

Hi, I have got a grok :

filter {
grok {
match => [
"message", "%{SYSLOGTIMESTAMP:date}%{SPACE}%{IPV4:clientip}%{SPACE}%{SYSLOGTIMESTAMP}%{SPACE}[%{WORD:onLocation}:%{WORD:onComponent}.%{WORD:onVolume}.%{WORD:onEvent}:%{WORD:reportedSeverity}]: Volume %{WORD:server}@vserver:%{UUID}%{SPACE}%{WORD:eventExecuted}%{GREEDYDATA:trash}"
]
}
}

which I run on this log :
"May 5 11:35:12 10.66.212.139 May 5 11:35:12 [derotnp00026:wafl.vvol.destroyed:inform]: Volume derotvi0131_test_SA1@vserver:7d760e7f-80d0-11e2-9ec0-123478563412 destroyed. ","2014-05-03T11:35:12.000+0200",11,3,35,may,12,saturday,2014,local,,"10.66.212.139","sw-syslog",1,,,"::...::[:..:]:__@:----._","udp:514",syslog,spwdfvml0703,,,,,,,,,,,,,"Volume derotvi0131_test_SA1@vserver:7d760e7f-80d0-11e2-9ec0-123478563412 destroyed.",info,,,,info,,,,,,,info,"Volume derotvi0131_test_SA1@vserver:7d760e7f-80d0-11e2-9ec0-123478563412 destroyed.",,,info,"wafl.vvol.destroyed:info",,,,15,,0

The grok works fine in grok debugger but for some strange reason logstash doesn't pick up the log. The --debug mode states the following:

closing {:plugin=><LogStash::Filters::Grok match=>{"message"=>"%{SYSLOGTIMESTAMP:date}%{SPACE}%{IPV4:clientip}%{SPACE}%{SYSLOGTIMESTAMP}%{SPACE}\[%{WORD:onLocation}\:%{WORD:onComponent}\.%{WORD:onVolume}\.%{WORD:onEvent}\:%{WORD:reportedSeverity}\]\: Volume %{WORD:server}\@vserver:%{UUID}%{SPACE}%{WORD:eventExecuted}%{GREEDYDATA:trash}"}, periodic_flush=>false, patterns_files_glob=>"*", break_on_match=>true, named_captures_only=>true, keep_empty_captures=>false, singles=>true, tag_on_failure=>["_grokparsefailure"]>, :level=>:debug, :file=>"logstash/plugin.rb", :line=>"35", :method=>"do_close"}

Am I doing anything wrong here.
Thanks

2

So you're saying Logstash doesn't appear to be processing the message at all? What does the rest of the configuration look like?

3

It is a very basic config :
input {
file {
path => ["/Users/alialaie/Desktop/Examples/Data/destroyed.log"]
start_position => "beginning"
sincedb_path => "/dev/null"
}
}

filter {
grok {
match => [
"message", "%{SYSLOGTIMESTAMP:date}%{SPACE}%{IPV4:clientip}%{SPACE}%{SYSLOGTIMESTAMP}%{SPACE}[%{WORD:onLocation}:%{WORD:onComponent}.%{WORD:onVolume}.%{WORD:onEvent}:%{WORD:reportedSeverity}]: Volume %{WORD:server}@vserver:%{UUID}%{SPACE}%{WORD:eventExecuted}%{GREEDYDATA:trash}"
]
}
}

output {
stdout {
codec => rubydebug
}
}

and the log file contains 2 lines just for test purposes :

"May 5 11:35:12 10.66.212.139 May 5 11:35:12 [derotnp00026:wafl.vvol.destroyed:inform]: Volume derotvi0131_test_SA1@vserver:7d760e7f-80d0-11e2-9ec0-123478563412 destroyed. ","2014-05-03T11:35:12.000+0200",11,3,35,may,12,saturday,2014,local,,"10.66.212.139","sw-syslog",1,,,"::...::[:..:]:__@:----.","udp:514",syslog,spwdfvml0703,,,,,,,,,,,,,"Volume derotvi0131_test_SA1@vserver:7d760e7f-80d0-11e2-9ec0-123478563412 destroyed.",info,,,,info,,,,,,,info,"Volume derotvi0131_test_SA1@vserver:7d760e7f-80d0-11e2-9ec0-123478563412 destroyed.",,,info,"wafl.vvol.destroyed:info",,,,15,,0
"Apr 30 15:06:08 10.66.212.73 Apr 30 15:06:08 [derotnp00002:wafl.vvol.destroyed:info]: Volume vol_Q4C_db1 destroyed. ","2014-04-30T15:06:08.000+0200",15,30,6,april,8,wednesday,2014,local,,"10.66.212.73","sw-syslog",1,,,"__::...__::[:..:]:_.","udp:514",syslog,spwdfvml0703,,,,,,,,,,,derotnp00002,,"Volume vol_Q4C_db1 destroyed.",info,,,,info,,,,derotnp00002,,,info,"Volume vol_Q4C_db1 destroyed.",,,info,"wafl.vvol.destroyed:info",,,,15,,0
"Apr 30 15:06:04 10.66.212.73 Apr 30 15:06:04

And nothing comes out of the stdout !

4

And you're running Logstash as yourself in a terminal?

5

Yes that's right.

Get Outlook for iOShttps://aka.ms/o0ukef

6

Weird. Trying starting it with --verbose to get additional clues about what the file input is doing.

7

Hi Magnus,
The outcome with --verbose is more than 5k characters ! How could I share this with you.
Thanks
Ali

8

Use pastebin or similar. But before you do that, look through it yourself and see if you can find references to sincedb and/or the path to the input file.