Multiline Codec Issues

Elastic Stack Logstash
#1

So for example, if I have a BASIC stack trace like this:

Caused by: java.lang.IllegalArgumentException: field [@timestamp] doesn't exist
at org.elasticsearch.action.fieldstats.TransportFieldStatsTransportAction.shardOperation(TransportFieldStatsTransportAction.java:166)

and this config

input {
some_input {
some_settings
codec => multiline {
source => "message"
pattern => "^\s"
what => "previous"
}
}
}
filter {
mutate {
gsub => [ "message", "field", "SOMESTRING" ]
}
}
output {
some_output { }
}

I understand that, the word "field" in the FIRST line would be substituted out for SOMESTRING, but when the second line enters your logstash pipeline and hits the multiline codec, it would return a match, and then be added to the previous event. Would that data then make it through the rest of the logstash pipeline?? Wouldn't the first event have passed through the mutate filter?? Would the data be appended to the message field?? I'm just not clear on how this interaction works. Thank you!!

#2

Pretend there was a tab in front of the "at" in the second line of that stack trace. Formatting problems. >:(

#3

I've moved this to the Logstash category for you :slight_smile:

1 Like
#4

Would that data then make it through the rest of the logstash pipeline??

Yes.

Wouldn't the first event have passed through the mutate filter??

No. The multiline buffers input physical lines until it concludes that the current event should be released.

Would the data be appended to the message field??

Yes. What you'll get in the end is one event with a message field containing "Caused by: java.lang.IllegalArgumentException: field [@timestamp] doesn't exist\n\tat org.elasticsearch.action.fieldstats.TransportFieldStatsTransportAction.shardOperation(TransportFieldStatsTransportAction.java:166)".

I'm not sure you have the best pattern though. Doesn't your log messages begin with a timestamp? The typical way to configure a multiline codec for log files with potentially multiline events is this:

multiline {
  pattern => "^some regexp that matches the timestamp"
  negate => true
  what => "previous"
}
#5