# Filebeat multiline not working with autodiscover **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [September 4, 2019, 8:56pm UTC](https://discuss.elastic.co/t/filebeat-multiline-not-working-with-autodiscover/198103 "2019-09-04T20:56:38Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![zaratustra689](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaratustra689/32/43118_2.png) [@zaratustra689](https://discuss.elastic.co/u/zaratustra689)\ **Post date:** [September 4, 2019, 8:56pm UTC](https://discuss.elastic.co/t/filebeat-multiline-not-working-with-autodiscover/198103/1 "2019-09-04T20:56:38Z")
Hello, I have the following configuration in `filebeat.yml` and the multiline feature is NOT working as [expected](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html#multiline). I have read previous posts with this issue, but the difference is that i'm NOT using prospectors or inputs. I'm using **autodiscover**. Single line events are working properly, however multiline events never show up in kibana. This is my `filebeat.yml`: ```auto filebeat.autodiscover: providers: - type: docker templates: - condition.contains: docker.container.image: "xxxxxx" config: - type: log paths: - "/var/lib/docker/containers/${data.docker.container.id}/*.log" json.keys_under_root: true json.add_error_key: true json.message_key: log multiline.pattern: '^.,' multiline.negate: true multiline.match: after exclude_lines: ['healthcheck'] encoding: utf-8 close_inactive: 5m setup.template.enabled: true setup.template.name: "xxxx" setup.template.pattern: "xxxx*" setup.kibana: host: "xxxx" processors: - add_host_metadata: netinfo.enabled: true - add_cloud_metadata: ~ - add_docker_metadata: ~ output.logstash: hosts: ["xxxxx"] xpack.monitoring: enabled: true elasticsearch: hosts: ["xxxxxx"] ``` Also, **logstash** is sending this log: ```auto [2019-09-04T19:07:44,750][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=> ["index", { :_id=>"2eaacaee422a4b52e5e7bbe926a9beac781d53b014face27edca98f4f9f83dc3", :_index=>"xxxxxxxxx", :_type=>"doc", :routing=>nil }, # ], :response=>{ "index"=>{ "_index"=>"xxxxxxxx", "_type"=>"doc", "_id"=>"2eaacaee422a4b52e5e7bbe926a9beac781d53b014face27edca98f4f9f83dc3", "status"=>400, "error"=>{ "type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [message] of type [text]", "caused_by"=>{ "type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:52" } } } } } ``` ---
**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\ **Post date:** [September 9, 2019, 6:10am UTC](https://discuss.elastic.co/t/filebeat-multiline-not-working-with-autodiscover/198103/2 "2019-09-09T06:10:39Z")
@zaratustra689, In you `filebeat.yml` not seen how you are managing multipline logs? Can you please provide some sample logs? Thanks. ---
**Author:** ![zaratustra689](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaratustra689/32/43118_2.png) [@zaratustra689](https://discuss.elastic.co/u/zaratustra689)\ **Post date:** [September 9, 2019, 5:52pm UTC](https://discuss.elastic.co/t/filebeat-multiline-not-working-with-autodiscover/198103/3 "2019-09-09T17:52:07Z")
@Tek_Chand You are right. I updated the `yml` above. 🙂 ---
**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\ **Post date:** [September 10, 2019, 4:36am UTC](https://discuss.elastic.co/t/filebeat-multiline-not-working-with-autodiscover/198103/4 "2019-09-10T04:36:31Z")
@zaratustra689, After managing multiple lines logs is your issue fixed? ---
**Author:** ![zaratustra689](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaratustra689/32/43118_2.png) [@zaratustra689](https://discuss.elastic.co/u/zaratustra689)\ **Post date:** [September 10, 2019, 2:38pm UTC](https://discuss.elastic.co/t/filebeat-multiline-not-working-with-autodiscover/198103/5 "2019-09-10T14:38:58Z")
@Tek_Chand No, that's the thing, I use the config above with the multiline settings (which I forgot to paste in the first post) and it does not work as expected, it does not even save the multiliine logs to elasticsearch. It just miss them and doesn't process them as a document with a multiline message log ---
**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\ **Post date:** [September 11, 2019, 4:39am UTC](https://discuss.elastic.co/t/filebeat-multiline-not-working-with-autodiscover/198103/6 "2019-09-11T04:39:01Z")
@zaratustra689, can you please provide few sample logs for multiline? That will be helpful. > [@zaratustra689](#): > > multiline.pattern: '^.,' From the above pattern it seems that your multiline log line is started with `,`. Please provide one or two sample log so we can extend the pattern and that may work. Thanks. ---
**Author:** ![zaratustra689](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaratustra689/32/43118_2.png) [@zaratustra689](https://discuss.elastic.co/u/zaratustra689)\ **Post date:** [September 13, 2019, 7:02pm UTC](https://discuss.elastic.co/t/filebeat-multiline-not-working-with-autodiscover/198103/7 "2019-09-13T19:02:34Z")
Hi all and thank you for your help @Tek_Chand. First of all I wanna add an example of my logs so that you can understand the multiline pattern I chose and configured according to [this official documentation](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html). ```auto I, [2019-09-13T18:47:21.452847 #24] INFO -- : [061ffeb0-2333-447c-ba84-c3346ed8444f, 127.0.0.1, someone@gmail.com] EXEC gobstones:test:2 - SCHED 0:24 ``` Now I have finally managed to get my **multiline logs** working with **docker autodiscover** and **filebeat** version **6.6.2**. My solution unfortunately implies **upgrading** from **filebeat 6.5.4** to **filebeat 6.6.2**. That is because I couldn't get it working in 6.5.4 but the same configuration in 6.6.2 works. So my final **filebeat.yml** autodiscover config is: ```auto filebeat.autodiscover: providers: - type: docker templates: - condition.contains: docker.container.image: "xxxx" config: - type: docker containers.ids: - "${data.docker.container.id}" exclude_lines: ['healthcheck'] # this is optional multiline.pattern: '^.,' multiline.negate: true multiline.match: after encoding: utf-8 close_inactive: 5m ``` I had to change in the **`template`** the `type: log` to `type: docker` and add the `containers.ids:` setting. I hope it helps ! Regards, Caro ---
**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\ **Post date:** [September 16, 2019, 3:45am UTC](https://discuss.elastic.co/t/filebeat-multiline-not-working-with-autodiscover/198103/8 "2019-09-16T03:45:13Z")
@zaratustra689, Glad to hear that you are able to fix this issue. You can use the below pattern also in `multiline.pattern` ```auto multiline.pattern: '[A-Z]{1}, \[[0-9]{4}-[0-9]{2}-[0-9]{2}' ``` Thanks. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 14, 2019, 3:45am UTC](https://discuss.elastic.co/t/filebeat-multiline-not-working-with-autodiscover/198103/9 "2019-10-14T03:45:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.