Filebeat multiline setting not working

[rituzy (Roman)]

Hi, I was trying to consolidate stack trace by filebeat as per instruction here

Manage multiline messages | Filebeat Reference [7.0] | Elastic

This is my filebeat.yml settings

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/myHomeFolder/log_example.log    
  fields:
    app: myapp
    multiline.pattern: '^Excep'
    multiline.negate: true
    multiline.match: after

  name: filebeat-myapp-test

tags: ["", "env-test"]

# Optional fields that you can specify to add additional information to the
# output.
fields:
  env: test
  app: myapp

This is an example log file /home/myHomeFolder/log_example.log :

Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

This is an output of command
sudo /usr/share/filebeat/bin/filebeat -e -c /home/myHomeFolder/filebeat.yml
(part of the output) :

. . .
{
"@timestamp": "2019-08-07T14:29:13.559Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.0.1"
},
"log": {
"file": {
"path": "/home/myHomeFolder/log_example.log"
},
"offset": 2821
},
"message": "Exception in thread "main" java.lang.NullPointerException",
"tags": [
"",
"env-test"
],
"input": {
"type": "log"
},
"fields": {
"env": "test",
"app": "myapp",
"multiline": {
"pattern": "^Excep",
"negate": true,
"match": "after"
}
},
"ecs": {
"version": "1.0.0"
},
"host": {
"name": "myapp-test"
},
"agent": {
"id": "0f1992b2-2dce-486c-a400-76cbe7e16a06",
"version": "7.0.1",
"type": "filebeat",
"ephemeral_id": "6a1ef472-992a-40ff-a448-04fa037e3bef",
"hostname": "myapp-test"
}
}
{
"@timestamp": "2019-08-07T14:29:13.559Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.0.1"
},
"message": " at com.example.myproject.Book.getTitle(Book.java:16)",
"tags": [
"",
"env-test"
],
"input": {
"type": "log"
},
"fields": {
"env": "test",
"app": "myapp",
"multiline": {
"pattern": "^Excep",
"negate": true,
"match": "after"
}
},
"agent": {
"id": "0f1992b2-2dce-486c-a400-76cbe7e16a06",
"version": "7.0.1",
"type": "filebeat",
"ephemeral_id": "6a1ef472-992a-40ff-a448-04fa037e3bef",
"hostname": "myapp-test"
},
"ecs": {
"version": "1.0.0"
},
"host": {
"name": "myapp-test"
},
"log": {
"offset": 2879,
"file": {
"path": "/home/myHomeFolder/log_example.log"
}
}
}
{
"@timestamp": "2019-08-07T14:29:13.559Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.0.1"
},
"message": " at com.example.myproject.Author.getBookTitles(Author.java:25)",
. . .

As you can see, it splits every message separately(every line one by one) But I want to gather all this staff for the stack trace in a one message and than send to logstash server.

I tried to switch multiline.negate to true or false and no luck

Can anyone help with this?

There is one error on filebeat starting
2019-08-07T17:28:23.554+0300 ERROR fileset/modules.go:125 Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
and one warning
2019-08-07T17:28:23.555+0300 WARN beater/filebeat.go:357 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
But I believe it does not affect at all...

Thanx in advance!

[justkind]

Can you try moving your multiline options up a level?

I don't think it should be under the fields option as it should be on the same level according to the log input documentation:
https://www.elastic.co/guide/en/beats/filebeat/7.0/filebeat-input-log.html

Ex.

fields:
  app: myapp
multiline.pattern: '^Excep'
multiline.negate: true
multiline.match: after

You can also look at a reference yml here and see how everything is setup:
https://www.elastic.co/guide/en/beats/filebeat/7.0/filebeat-reference-yml.html

Regarding your multiline pattern, you can test it on the Go Playground, Filebeat has documentation on this at: https://www.elastic.co/guide/en/beats/filebeat/7.0/_test_your_regexp_pattern_for_multiline.html

According to the playground, your pattern seems to do what you want it to do so if you can get the multiline options to work everything should fall into place

[rituzy (Roman)]

Bingo!
It was the root problem, now it does as expected:

{
  "@timestamp": "2019-08-08T07:59:02.414Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.0.1"
  },
  "message": "Exception in thread \"main\" java.lang.NullPointerException\n        at com.example.myproject.Book.getTitle(Book.java:16)\n        at com.example.myproject.Author.getBookTitles(Author.java:25)\n        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)\n",
  "log": {
    "offset": 0,
    "file": {
      "path": "/home/myHomeFolder/log_example.log"
    },
    "flags": [
      "multiline"
    ]
  },

Thank you so much for you prompt fruitful reply!