Logstash Grok New Line Syntax

Elastic Stack Logstash
1

Hi,

I am attempting to parse log data that are on different lines. When its read through filebeat and sent to logstash, the message contains \n. Unfortunetly, I am unable to deal with this and am getting an error when I parse it like this:

59%20PM
Screen Shot 2019-07-24 at 5.58.59 PM.png1058×460 33.3 KB

2

Use a literal newline in the configuration to match a multiline pattern, like this.

3

51%20AM
Screen Shot 2019-07-25 at 10.28.51 AM.png1040×418 17.4 KB

The following doesn't work either. Is that what you meant?

4

If your events contain newline then you need a literal newline in the pattern. However, if your events contain \n then you would use \\n.

5

My event contains a literal newline. However, when I output my logstash output to the console, I see \n in the "message" field.
I just adjusted my config syntax to this:

grok {
      match => [
        "message", "(?<logdate>[A-Z]{1}[a-z]{2} [0-9]{2}, [0-9]{4} [0-9]{1}:[0-9]{2}:[0-9]{2} [A-Z]{2}) %{NOTSPACE:requestpath} %{WORD:word}                                                                    
        %{WORD:status}%{GREEDYDATA:data}",

This unfortunately does not correctly parse the data.

6

Shouldn't that be

    grok {
        match => [ "message", "(?<logdate>[A-Z]{1}[a-z]{2} [0-9]{2}, [0-9]{4} [0-9]{1}:[0-9]{2}:[0-9]{2} [A-Z]{2}) %{NOTSPACE:requestpath} %{WORD:word} 
%{WORD:status}%{GREEDYDATA:data}",

?

7

I believe either format works. Just adjusted to your format and it is still not being parsing correctly. However, I tried the //n and am having a tricky issue:

47%20AM
Screen Shot 2019-07-25 at 11.27.47 AM.png1059×702 72.5 KB

However, this parses correctly if I take out status:

48%20AM
Screen Shot 2019-07-25 at 11.28.48 AM.png1069×686 61.1 KB

edit: And when I do the following, I get a grokparsefailure:

"message", "(?<logdate>[A-Z]{1}[a-z]{2} [0-9]{2}, [0-9]{4} [0-9]{1}:[0-9]{2}:[0-9]{2} [A-Z]{2}) %{NOTSPACE:requestpath} %{WORD:word}\\n%{GREEDYDATA:data}"
It parses correctly up until the \n. So while the grok debugger is showing that it should be parsing correctly, it actually is not occuring when I run logstash :frowning:

8

30%20PM
Screen Shot 2019-07-25 at 12.43.30 PM.png1093×724 73.4 KB

This is also another configuration that works in the debugger, but not when I run logstash

9

What is the text you are trying to match against?

10

Mar 05, 2019 3:23:14 AM pluggin word\nWARNING: datadata

I ended up using gsub to drop "\n" and replace it with a space. From there, I was able to configure a pattern that worked.
"message", "(?<logdate>[A-Z]{1}[a-z]{2} [0-9]{2}, [0-9]{4} [0-9]{1}:[0-9]{2}:[0-9]{2} [A-Z]{2}) %{NOTSPACE:requestpath} %{WORD:word} %{WORD:status}: %{GREEDYDATA:data}",

Still a little weird as to why the debugger would say it was parsed correctly while logstash would throw a grokparsefailure error.

Anyways, appreciate the help. Here is the mutate gsub I used for reference.
mutate {gsub =>["message", "[\n]"," "]}

11

Do you have config.support_escapes enabled?

Anyways, I think the problem was the trailing space at the end of the first line of your pattern.

12

That seems to be the issue, I had it set to false.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.