Logstash Grok New Line Syntax

[userR]

Hi,

I am attempting to parse log data that are on different lines. When its read through filebeat and sent to logstash, the message contains \n. Unfortunetly, I am unable to deal with this and am getting an error when I parse it like this:

[Badger]

Use a literal newline in the configuration to match a multiline pattern, like this.

[userR]

The following doesn't work either. Is that what you meant?

[Badger]

If your events contain newline then you need a literal newline in the pattern. However, if your events contain \n then you would use \\n.

[userR]

My event contains a literal newline. However, when I output my logstash output to the console, I see \n in the "message" field.
I just adjusted my config syntax to this:

grok {
      match => [
        "message", "(?<logdate>[A-Z]{1}[a-z]{2} [0-9]{2}, [0-9]{4} [0-9]{1}:[0-9]{2}:[0-9]{2} [A-Z]{2}) %{NOTSPACE:requestpath} %{WORD:word}                                                                    
        %{WORD:status}%{GREEDYDATA:data}",

This unfortunately does not correctly parse the data.

[Badger]

Shouldn't that be

    grok {
        match => [ "message", "(?<logdate>[A-Z]{1}[a-z]{2} [0-9]{2}, [0-9]{4} [0-9]{1}:[0-9]{2}:[0-9]{2} [A-Z]{2}) %{NOTSPACE:requestpath} %{WORD:word} 
%{WORD:status}%{GREEDYDATA:data}",

?

[userR]

I believe either format works. Just adjusted to your format and it is still not being parsing correctly. However, I tried the //n and am having a tricky issue:

However, this parses correctly if I take out status:

edit: And when I do the following, I get a grokparsefailure:

"message", "(?<logdate>[A-Z]{1}[a-z]{2} [0-9]{2}, [0-9]{4} [0-9]{1}:[0-9]{2}:[0-9]{2} [A-Z]{2}) %{NOTSPACE:requestpath} %{WORD:word}\\n%{GREEDYDATA:data}"
It parses correctly up until the \n. So while the grok debugger is showing that it should be parsing correctly, it actually is not occuring when I run logstash

[userR]

This is also another configuration that works in the debugger, but not when I run logstash

[Badger]

What is the text you are trying to match against?

[userR]

Mar 05, 2019 3:23:14 AM pluggin word\nWARNING: datadata

I ended up using gsub to drop "\n" and replace it with a space. From there, I was able to configure a pattern that worked.
"message", "(?<logdate>[A-Z]{1}[a-z]{2} [0-9]{2}, [0-9]{4} [0-9]{1}:[0-9]{2}:[0-9]{2} [A-Z]{2}) %{NOTSPACE:requestpath} %{WORD:word} %{WORD:status}: %{GREEDYDATA:data}",

Still a little weird as to why the debugger would say it was parsed correctly while logstash would throw a grokparsefailure error.

Anyways, appreciate the help. Here is the mutate gsub I used for reference.
mutate {gsub =>["message", "[\n]"," "]}

[Badger]

Do you have config.support_escapes enabled?

Anyways, I think the problem was the trailing space at the end of the first line of your pattern.

[userR]

That seems to be the issue, I had it set to false.