Grok pattern to match Filebeat multiline input up to the first new line character

Elastic Stack Logstash
1

Logstash 6.8.10
Filebeat 6.8.9

Filebeat config:

filebeat.inputs:

- type: log
  enabled: true
  paths:
  - '/path/to/file'
  tags:
  - 'app'
  fields:
    app_id: some_app
  multiline.pattern: '^\[[0-9]{4}\-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

logging.level: info
logging.metrics.period: 10m
logging.to_files: true
logging.files:
  keepfiles: 4
  permissions: 0644

Filebeat is sending Logstash data with message fields like this:

"message": "[2020-10-28 15:27:24,646] ERROR - SomethingSomething.otherThing(SomethingSomething.java:85) - Exception while doing something\njava.lang.NullPointerException\notherMethodCalls"

Now, in my Logstash setup I'd like to parse everything up to the first new line character, basically just the first log line.

What I thought about using was this:

filter {
    grok {
        match => { "message" => "^\[%{TIMESTAMP_ISO8601:[@metadata][logtime]}\]%{SPACE}%{LOGLEVEL:level}%{SPACE}-%{SPACE}%{DATA:classInfo}%{SPACE}-%{SPACE}%{GREEDYDATA:[@metadata][messageBody]}$" }
    }
}

But this for some reason does not work:

{
  "classInfo": "SomethingSomething.otherThing(SomethingSomething.java:85)",
  "level": "ERROR",
  "[@metadata][logtime]": "2020-10-28 15:27:24,646",
  "[@metadata][messageBody]": "Exception while doing something\njava.lang.NullPointerException\notherMethodCalls"
}

Afaik GREEDYDATA should not match new line unless the (?m) flag is given. So what am I getting wrong here? Thanks!

2

Change GREEDYDATA to DATA and you will get

 "@metadata" => {
        "logtime" => "2020-10-28 15:27:24,646",
    "messageBody" => "Exception while doing something"
},
3

Thanks! So GREEDYDATA does match newline (\n) by default? Regardless if I use the (?m) flag or not?

4

It appears to, yes.

5

Really odd since GREEDYDATA is .* I think and the Oniguruma regex docs say that . any character (except newline) (link).

6

If anyone could point out if there is a default in Logstash where dot matches newlines, I'd be really thankful!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.