Problem with coding Grok filter

[ibrahimsharaf (Ibrahim Sharaf El Den)]

Hi there, I have this message
2017-05-01 08:09:19 [scraper.py] ERROR: Error downloading <POST http://localhost:8050/render.html>

I want to write a grok filter to extract this part Error downloading <POST http://localhost:8050/render.html>

I coded this'ERROR': %{GREEDYDATA:error_msg} but it doesn't match, can you help?

[ugosan (Ugo Sangiorgi)]

@ibrahimsharaf It would be something like this:
%{TIMESTAMP_ISO8601}%{SPACE}%{NOTSPACE}%{SPACE}%{LOGLEVEL}%{NOTSPACE}%{SPACE}%{GREEDYDATA:error_msg}

[ibrahimsharaf (Ibrahim Sharaf El Den)]

@ugosan How can I get only the ERROR messages? I don't want to get other loglevel messages such as DEBUG and INFO.

[Nico-DF]

You can use:

%{TIMESTAMP_ISO8601} %{NOTSPACE} ERROR: %{GREEDYDATA:error_msg}

and then inside grok filter, add the loglevel field, or either:

%{TIMESTAMP_ISO8601} %{NOTSPACE} (?<Loglevel>ERROR): %{GREEDYDATA:error_msg}

The synthax with (? ) defines a field with < name > that match the regex pattern that follows. And a plain string is a also a regex.

With any of these pattern, you will only get messages with loglevel ERROR (case sensitive).

[ugosan (Ugo Sangiorgi)]

@ibrahimsharaf Well if you tag your LOGLEVEL with a variable name, say loglevel like:
%{TIMESTAMP_ISO8601}%{SPACE}%{NOTSPACE}%{SPACE}%{LOGLEVEL:loglevel}%{NOTSPACE}%{SPACE}%{GREEDYDATA:error_msg}

...then you might filter it out before sending it to elasticsearch like:

output {
      if [loglevel] == 'ERROR' {
             elasticsearch { 
                     ... 
             }
      }
}