Grok Pattern not parsing

Elastic Stack Logstash
1

Hi, I'm trying to parse a log file with content like this:

2016-05-09 12:00:00,006 INFO  [com.level2.quartz.BaseLevel2EngineJob] (MVCScheduler_Worker-1) Executing job It removes expired activation codes
2016-05-09 12:00:00,006 INFO  [com.seglan.mvc.batch.MvcActivationCodesJobImpl] (MVCScheduler_Worker-1) Activation codes cleanup job is running...

I've tested the filter using the Grok Debugger but when starting logstash, nothing is parsed.

Here is my logstash.conf file:

input {
  file {
        path => "/home/dlopez/server.log"
        start_position => "beginning"
       }
}

filter {
   grok {
     match =>
        {
          "message" => "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:level}\s+\[%{DATA:className}\]%{SPACE}%{GREEDYDATA:message}"
        }
   }
}
output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "logstash-%{+YYYY.MM.dd}"
  }
}

Is there any way to test the pattern?

Any help would be appreciated. Thanks

Regards

2

Comment out the elasticsearch output and replace it with stdout { codec => rubydebug } to shorten the feedback loop. You might also find https://github.com/magnusbaeck/logstash-filter-verifier useful.

3

Thank you very much!

this is what I got:

{
"message" => "2016-05-09 12:00:00,029 DEBUG [com.googlecode.genericdao.search.BaseSearchProcessor] (MVCScheduler_Worker-1) generateQL:",
"@version" => "1",
"@timestamp" => "2016-05-11T12:40:49.362Z",
"path" => "/home/dlopez/server_test.log",
"host" => "sgl-v6-hce-piraeusbank-back",
"tags" => [
[0] "_grokparsefailure"
]
}

My parser is not working fine but I don't know why

Thanks

4

Start small and increase the complexity. Begin with the simplest possible expression (%{TIMESTAMP_ISO8601:timestamp}) and make sure that works. Then add more and more tokens until things break.

1 Like
5

You can test your patterns with this debugger:

http://grokdebug.herokuapp.com/

6

Thanks, I've used that debugger to compose and test my pattern but unfortunately once I've moved into the system it doesn't work

7

Maybe something like this?

%{TIMESTAMP_ISO8601:timestamp} %{WORD:level}\s*\[(?<className>[A-Z,a-z,.,0-9]*)] %{GREEDYDATA:message}

8

Ok, it seems the problem was here:

\s+[%{DATA:className}]

I'm trying to parse the "class name" part of the message, I've tried the following as well but with no luck:

%{TIMESTAMP_ISO8601:timestamp}%{SPACE} %{LOGLEVEL:level} %{SPACE} %{SPACE} \[%{JAVACLASS:class}\] %{SPACE} %{GREEDYDATA:message}

Thanks

9

Well, finally I got it working:

{ "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} \s+[%{JAVACLASS:class}] %{GREEDYDATA:message} " }

I removed all the ${SPACE} and now it can be parsed

Thanks a lot for your help and clues!