Multiline Codec logstash for log level

Elastic Stack Logstash
1

Hi
I have a log entry sample as below. every line starts with i.e., log level.
Few log entries would be on the same line but few would be in multiple lines.
Can you please suggest the logstash file input multiline code pattern for this.

Sample:
[TRACE] [2019-07-31 10:31:38,889] ...................................................................................................................................................</SOAP-ENV:Body></SOAP-ENV:Envelope>]

[INFO] [2019-07-31 10:31:38,889] ...................................................................................................................................................</SOAP-ENV:Body></SOAP-ENV:Envelope>]

2

Try

 multiline { pattern => "^\[%{LOGLEVEL}\]" negate => true what => previous auto_flush_interval => 1 }
3

Hi
Unfortunately I realized that the file log line starts with some anscii color sequences/ESC characters as below

ESC[mESC[32m[INFO ][2019-07-31 10:31:38,889]................................................................

Please suggest what should be the pattern

Note: The logstash grok parse failure log shows the above ESC characters as below
\u001B[m\u001B[35m[INFO ][2019-07-31 10:31:38,889]................................................................

Thanks

4

I am not feeling up to writing a regexp that matches arbitrary ISO 6429 sequences, but if all you need to handle is SGRs then this would work

multiline { pattern => "^(^[\[[0-9]*m)*\[%{LOGLEVEL}\]" negate => true what => previous auto_flush_interval => 1 } }

Note that the ^[ is a literal escape character. That results in

   "message" => "\e[m\e[32m[TRACE] [2019-07-31 10:31:38,889] ...................................................................................................................................................</SOAP-ENV:Body></SOAP-ENV:Envelope>]",
5

Got the following error from logstash and exited.

[2019-08-01T04:35:39,283][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"RegexpError", :message=>"premature end of char-class: /^(^[\[[0-9]m)\[(?([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?))\]/m", :backtrace=>["org/jruby/RubyRegexp.java:940:in initialize'", "/home/hybrisuser/elk/logstash-7.2.0/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:127:incompile'", "/home/hybrisuser/elk/logstash-7.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-codec-multiline-3.0.10/lib/logstash/codecs/multiline.rb:166:in register'", "/home/hybrisuser/elk/logstash-7.2.0/logstash-core/lib/logstash/codecs/base.rb:18:ininitialize'", "org/logstash/plugins/PluginFactoryExt.java:258:in plugin'", "org/logstash/execution/JavaBasePipelineExt.java:50:ininitialize'", "/home/hybrisuser/elk/logstash-7.2.0/logstash-core/lib/logstash/java_pipeline.rb:24:in initialize'", "/home/hybrisuser/elk/logstash-7.2.0/logstash-core/lib/logstash/pipeline_action/create.rb:36:inexecute'", "/home/hybrisuser/elk/logstash-7.2.0/logstash-core/lib/logstash/agent.rb:325:in `block in converge_state'"]}

6

That suggests that you have entered ^[ as a caret and a square bracket instead of a literal escape character (ctrl/v+esc on UNIX).

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.