Logstash multiline codec functional question

Elastic Stack Logstash
1

Hello,

I am kind of new to logstash and was wondering about the functionality of the multiline codec.

If I have a multiline codec that has the following definition:

input {
beats {
port => "5045"
codec => multiline {
multiline_tag => "MULTILINE"
patterns_dir => "C:/dev/elk/logstash-5.4.0/custom_patterns"
pattern => "SOME_MULTILINE_PATTERN"
what => "next"
}
}
}

All logging lines/events that pass the pattern are concatenated to form a single event. But the last line which does not match is also concatented.

Is that the expected functionity?

I would expect that if it does not match it is dropped from the multiline event and sent in the next event.

I was also wondering if it is possible to have more than one multiline codec definition. One that does the next line processing and another the previous line?

Regards Benny

2

Hello,

I was wondering if my question was clear?

Regards benny

3

All logging lines/events that pass the pattern are concatenated to form a single event. But the last line which does not match is also concatented.

I'm not sure exactly what you mean (examples help) but it since the last matching line joins with the next line what you describe sounds like the intended behavior. Perhaps you should use what => "previous" instead? Again, an example of what you're trying to do would make it easier to help.

I was also wondering if it is possible to have more than one multiline codec definition. One that does the next line processing and another the previous line?

When would you want to do this?

4

Hi Magnus,

Thank you for the reply!

Below is an example of my multiline pattern:

input {
beats {
port => "5045"
codec => multiline {
multiline_tag => "MULTILINE"
pattern => "(^111)|(^222)"
what => "next"
}
}
}

The output is:

{
"message" => "000",
"type" => "log",
"tags" => [
[0] "beats_input_codec_multiline_applied"
]
}
{
"message" => "111\n222\n333",
"type" => "log",
"tags" => [
[0] "MULTILINE",
[1] "beats_input_codec_multiline_applied"
]
}
{
"message" => "444",
"type" => "log",
"tags" => [
[0] "beats_input_codec_multiline_applied"
]
}

As you can see the 333 does not match 111 or 222 but is still concatenated.
If this is an (important) event I would not expect it to be swallowed in the prevoius event!

Is my example clear?

In reply to your other question.
I would like to do this in the input of logstach. This would give a clearer and simpler expression to match.
I have tested using two multiline expressions but as soon as the filrst multiline fails the second is never executed.

Regards Benny

5

As you can see the 333 does not match 111 or 222 but is still concatenated.
If this is an (important) event I would not expect it to be swallowed in the prevoius event!

Is my example clear?

It doesn't explain what your actual log entries look like and what rules should be used to join them. As I said you should probably use what => "previous" instead. In most cases multiline setups are based on what identifies the beginning of a logical event:

multiline {
  pattern => "regexp matching the first line of a multiline group"
  negate => true
  what => "previous"
}

I would like to do this in the input of logstach. This would give a clearer and simpler expression to match.
I have tested using two multiline expressions but as soon as the filrst multiline fails the second is never executed.

I don't understand. What did you test?

6

Hi Magnus,

What the actual log entries look like is not really the point.
I just gave a simplified example to show what I am expereincing using simplied rules.

In the mulitline output you see "message" => "111\n222\n333"

Sorry for repeating myself again but as you can see from the pattern ( pattern => "(^111)|(^222)" ) 333 should not be matched.
I also understand I have what => "next" in my expression.

The question is, should that event starting with 333 be part of the multiline event even though it does not match the pattern?
Or is this an error in the multiline codec or logstash?

Regards benny

7

Hi Benny,
let me try to explain the behavior of your config.

pattern => "(^111)|(^222)"
what => "next"

Your config says: Any line starting with 111 or 222 should be merged with next line that does not match the pattern.

With input 111\n222\n333.
111 comes -> 111 matches pattern -> logstash waits for next line
222 comes -> 222 matches pattern -> logstash waits for next line
333 comes -> 333 does not match -> all previous lines belongs to this line, logstash joins them and the event is completed.

8

Hi Oozza,

Got it. Thank you for the explaination!

Regards benny

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.