Filebeat multiline not working with autodiscover

[zaratustra689 (Carolina Baldino)]

Hello, I have the following configuration in filebeat.yml and the multiline feature is NOT working as expected.

I have read previous posts with this issue, but the difference is that i'm NOT using prospectors or inputs. I'm using autodiscover.

Single line events are working properly, however multiline events never show up in kibana.
This is my filebeat.yml:

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition.contains:
            docker.container.image: "xxxxxx"
          config:
            - type: log
              paths:
                - "/var/lib/docker/containers/${data.docker.container.id}/*.log"
              json.keys_under_root: true
              json.add_error_key: true
              json.message_key: log
              multiline.pattern: '^.,'
              multiline.negate: true
              multiline.match: after
              exclude_lines: ['healthcheck']
              encoding: utf-8
              close_inactive: 5m

setup.template.enabled: true
setup.template.name: "xxxx"
setup.template.pattern: "xxxx*"

setup.kibana:
  host: "xxxx"

processors:
  - add_host_metadata:
      netinfo.enabled: true
  - add_cloud_metadata: ~
  - add_docker_metadata: ~

output.logstash:
  hosts: ["xxxxx"]

xpack.monitoring:
  enabled: true
  elasticsearch:
    hosts: ["xxxxxx"]

Also, logstash is sending this log:

[2019-09-04T19:07:44,750][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. 
{:status=>400,
 :action=>
          ["index",
           {
            :_id=>"2eaacaee422a4b52e5e7bbe926a9beac781d53b014face27edca98f4f9f83dc3", 
            :_index=>"xxxxxxxxx", 
            :_type=>"doc",  
            :routing=>nil
           },  
           #<LogStash::Event:0x684e8b88>
          ], 
          :response=>{
                 "index"=>{
                     "_index"=>"xxxxxxxx", 
                     "_type"=>"doc", 
                     "_id"=>"2eaacaee422a4b52e5e7bbe926a9beac781d53b014face27edca98f4f9f83dc3", 
                     "status"=>400, 
                     "error"=>{
                            "type"=>"mapper_parsing_exception", 
                            "reason"=>"failed to parse field [message] of type [text]", 
                            "caused_by"=>{
                                    "type"=>"illegal_state_exception", 
                                   "reason"=>"Can't get text on a START_OBJECT at 1:52"
                             }
                       }
                    }
                 }
}

[Tek_Chand]

@zaratustra689, In you filebeat.yml not seen how you are managing multipline logs? Can you please provide some sample logs?

Thanks.

[zaratustra689 (Carolina Baldino)]

@Tek_Chand You are right. I updated the yml above.

[Tek_Chand]

@zaratustra689, After managing multiple lines logs is your issue fixed?

[zaratustra689 (Carolina Baldino)]

@Tek_Chand No, that's the thing, I use the config above with the multiline settings (which I forgot to paste in the first post) and it does not work as expected, it does not even save the multiliine logs to elasticsearch. It just miss them and doesn't process them as a document with a multiline message log

[Tek_Chand]

@zaratustra689, can you please provide few sample logs for multiline? That will be helpful.

@zaratustra689

multiline.pattern: '^.,'

From the above pattern it seems that your multiline log line is started with ,. Please provide one or two sample log so we can extend the pattern and that may work.

Thanks.

[zaratustra689 (Carolina Baldino)]

Hi all and thank you for your help @Tek_Chand.

First of all I wanna add an example of my logs so that you can understand the multiline pattern I chose and configured according to this official documentation.

I, [2019-09-13T18:47:21.452847 #24]  INFO -- : [061ffeb0-2333-447c-ba84-c3346ed8444f, 127.0.0.1, someone@gmail.com] EXEC gobstones:test:2 - SCHED 0:24

Now I have finally managed to get my multiline logs working with docker autodiscover and filebeat version 6.6.2.

My solution unfortunately implies upgrading from filebeat 6.5.4 to filebeat 6.6.2. That is because I couldn't get it working in 6.5.4 but the same configuration in 6.6.2 works.

So my final filebeat.yml autodiscover config is:

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition.contains:
            docker.container.image: "xxxx"
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
              exclude_lines: ['healthcheck'] # this is optional
              multiline.pattern: '^.,'
              multiline.negate: true
              multiline.match: after
              encoding: utf-8
              close_inactive: 5m

I had to change in the template the type: log to type: docker and add the containers.ids: setting.

I hope it helps !

Regards,
Caro

[Tek_Chand]

@zaratustra689, Glad to hear that you are able to fix this issue.

You can use the below pattern also in multiline.pattern

multiline.pattern: '[A-Z]{1}, \[[0-9]{4}-[0-9]{2}-[0-9]{2}'

Thanks.