Multiline parser filebeat 6.4.2

Hi there,

I am trying to group multiple lines as a single event depending upon timestamp. I am using Filebeat --> ElasticSearch --> Kibana (NO Logstash in between).To begin with i started manipulating the existing plugins (apache2 to be specific). I enabled it, modified the apache2/error/manifest.yml to point to the right directory from where to collect the logs, modified the _ingest/pipeline/filebeat-6.4.2-apache2-error-pipeline with my custom pipeline which contains grok and some more processors. Everything seemed to be working, but only collecting single line as an event. So i started looking around, and found i need to config multiline parser in filebeat.

So i went ahead, and added below lines in the apache2/error/config/error.yml(I am not sure if this is the correct file i need to put configuration in ? )


type: log
{{ range $i, $path := .paths }}
 - {{$path}}
{{ end }}
exclude_files: [".gz$"]
 pattern: "^\\[[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}(\\+|\\-)?[0-9]{2}:[0-9]{2}\\]"
 negate: true
 match: after

Below is my grok pattern in pipeline(it's collecting timestamp, serverId, error.level correctly)


Just pasting my grok processor, not the complete pipeline

      "grok": {
        "field": "message",
        "patterns": [
        "pattern_definitions" : {

Sample log :
[2018-10-23T00:17:13.064+00:00] [server_1] [WARNING] Exception in thread "main" java.lang.NullPointerException
at AnotherClassLoader.loadClass(
at test.main(
at Main.main(

[2018-10-23T00:17:13.438+00:00] [server_1] [ERROR] [] Missing https proxy settings.

So ideally, i would like to see 2 events / documents in ES, but it's not happening.

I am using filebeat 6.4.2

Any help is really appreciated.

Can anyone provide some feedback please ?

This look like the same use case that we have in our documentation, did you try with the following

multiline.pattern: '^\['
multiline.negate: true
multiline.match: after

This should create a new events every time the events start with [

Thanks a lot @pierhugues, yes i did try that and it worked but brought some unexpected behaviour as there were some messages which also had [ in the beginning of line, that's why i wanted to rely more on timestamp. I know, i am missing something very basic in my regex.

Oh k got it working, below is the regex, if someone looks in the future.


1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.