Parsing special charaters with grok

Elastic Stack Logstash
2

These codes are special ANSI sequences used to change the color in terminal output.

In simple words, these marks indicate to compatible console tools:
start block of green characters INFO (end of the block and) return to default color

Both the ^[ and the \e are different ways to encode a escape sequence; a good link to learn about it: http://jafrog.com/2013/11/23/colors-in-terminal.html

I don't know why filebeat uses a different way to represent the same character than your original console logs.

Some considerations about the regex pattern used for matching the special character in grok:

  • Setting the ^[ escape character is tricky, you can't just write the two characters "caret" and "square bracket".
    See Multiline Codec logstash for log level and https://stackoverflow.com/a/33479939

  • Have you tried \e instead of ^[ (something like \e\[32mINFO ... ) ?

  • You can also test another way to capture this special character with a regex: \x1B instead of either \e or ^[

Although it is not what you ask for, I will add a solution that works for my use cases; maybe it fits your purpose.

Usually I don't have control about which colors or formats will be used by the application logs in the systems that I work with. Now you are parsing green INFO messsages but ERROR ones will probably use red. Who knows what formats will be used in the future for other information...

When ANSI formatting is used, I prefer to strip the whole ANSI sequences before any other parsing (groks and similar filters). This makes the latter more readable and robust.

mutate {
  id => "[Meaningful label for your project, not repeated in any other config files] remove ANSI color codes"
  gsub => ["message", "\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]", ""]
}
1 Like