Multiline parser filebeat 6.4.2

[lukes]

Hi there,

I am trying to group multiple lines as a single event depending upon timestamp. I am using Filebeat --> ElasticSearch --> Kibana (NO Logstash in between).To begin with i started manipulating the existing plugins (apache2 to be specific). I enabled it, modified the apache2/error/manifest.yml to point to the right directory from where to collect the logs, modified the _ingest/pipeline/filebeat-6.4.2-apache2-error-pipeline with my custom pipeline which contains grok and some more processors. Everything seemed to be working, but only collecting single line as an event. So i started looking around, and found i need to config multiline parser in filebeat.

So i went ahead, and added below lines in the apache2/error/config/error.yml(I am not sure if this is the correct file i need to put configuration in ? )

apache2/error/config/error.yml

type: log
paths:
{{ range $i, $path := .paths }}
 - {{$path}}
{{ end }}
exclude_files: [".gz$"]
multiline:
 pattern: "^\\[[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}(\\+|\\-)?[0-9]{2}:[0-9]{2}\\]"
 negate: true
 match: after

Below is my grok pattern in pipeline(it's collecting timestamp, serverId, error.level correctly)

_ingest/pipeline/filebeat-6.4.2-apache2-error-pipeline

Just pasting my grok processor, not the complete pipeline

{
      "grok": {
        "field": "message",
        "patterns": [
          "\\[%{TIMESTAMP_ISO8601:msg.timestamp}\\]%{SPACE}\\[%{DATA:serverId}\\]%{SPACE}\\[(%{CUSTOM_LOGLEVEL:error.level})?\\]%{SPACE}%{GREEDYDATA:error.message}"
        ],
        "pattern_definitions" : {
          "CUSTOM_LOGLEVEL" : "%{LOGLEVEL}|NOTIFICATION"
        }
      }

Sample log :
[2018-10-23T00:17:13.064+00:00] [server_1] [WARNING] Exception in thread "main" java.lang.NullPointerException
at AnotherClassLoader.loadClass(test.java:58)
at test.main(test.java:30)
at Main.main(Main.java:68)

]]
[2018-10-23T00:17:13.438+00:00] [server_1] [ERROR] [] Missing https proxy settings.

So ideally, i would like to see 2 events / documents in ES, but it's not happening.

I am using filebeat 6.4.2

Any help is really appreciated.

[lukes]

Can anyone provide some feedback please ?

[pierhugues (Pier-Hugues Pellerin)]

This look like the same use case that we have in our documentation, did you try with the following

multiline.pattern: '^\['
multiline.negate: true
multiline.match: after

This should create a new events every time the events start with [

[lukes]

Thanks a lot @pierhugues, yes i did try that and it worked but brought some unexpected behaviour as there were some messages which also had [ in the beginning of line, that's why i wanted to rely more on timestamp. I know, i am missing something very basic in my regex.

[lukes]

Oh k got it working, below is the regex, if someone looks in the future.

^\[[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}(\+|\-)[0-9]{2}:[0-9]{2}\]