# Multiline issue with filebeat **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [July 22, 2020, 4:15pm UTC](https://discuss.elastic.co/t/multiline-issue-with-filebeat/242234 "2020-07-22T16:15:47Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![bryand](https://avatars.discourse-cdn.com/v4/letter/b/a87d85/32.png) [@bryand](https://discuss.elastic.co/u/bryand)\ **Post date:** [July 22, 2020, 4:15pm UTC](https://discuss.elastic.co/t/multiline-issue-with-filebeat/242234/1 "2020-07-22T16:15:47Z")
Hi, I'm having issues with trying to aggregate java exceptions into a single message using the filebeat multiline support. From everything I've read this should be very simple, yet, no matter what I try, it doesn't work. I've spent a few hours reading through similar issues that people have had, but none of the suggestions made for those, have worked for me. My setup is; ` filebeat -> logstash -> elasticsearch` all logstash is doing in this setup is directing the message to a particular elasticsearch index depending on the values of some metadata tags that are added by filebeat. I should add that we are running multiple services and each have their own instance of filebeat running - we are running on ubuntu 16.04, and filebeat is controlled by systemd. For each of the services, the multiline config is; ```auto multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}' multiline.negate: true multiline.match: after ``` where the log line begins with the date. An example of the logline is; ```auto 2020-07-22 15:41:24,174 - [WARN] - o.a.k.c.n.Selector - Error in I/O with hostA java.io.EOFException: null at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62) ~[service1.jar:1.0] at org.apache.kafka.common.network.Selector.poll(Selector.java:248) ~[service1.jar:1.0] at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) [service1.jar:1.0] at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) [service1.jar:1.0] at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) [service1.jar:1.0] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_211] ``` I have run this regexp through the play sandbox ([https://play.golang.org/](https://play.golang.org/)) with the above logline, and it matches fine. I have even tried variations on the regexp, as follows ```auto #multiline.pattern: '^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]' #multiline.pattern: '^\d?\d.\d\d.\d\d' ``` and while both return matches in the sandbox, I see no difference in the behaviour when I try them with my service. I have also tested my config with filebeat, as follows; ```auto root@:/# /home/filebeat/filebeat/filebeat test config config/filebeat.service1.yml Config OK ``` and I have used a yaml validator to ensure there are no formatting issues with my config file. At this point I am not sure what else I can do to pinpoint what my issue is. Firstly, can I discount logstash as being an issue here? The logstash config for this service does very little. It simply takes in the message and outputs it to a certain index. The logstash config is ```auto input { beats { port => 5044 } } filter { if "service1" in [tags]{ grok { match => {"message" => "INFO\s+\[%{TIMESTAMP_ISO8601:logtimestamp}\] %{GREEDYDATA:message}"} } } } output { if "service1" in [tags]{ elasticsearch { hosts => ["http://:9200"] index => "-%{+YYYY.MM.dd}" } } } ``` My own feeling is that logstash is not an issue and that I need to focus on the filebeat side. Unfortunately the logs are not terribly verbose and there is nothing to indicate whether my logline is being aggregated into a single message, or not. One thing that did occur to me is that there could be some hidden character in the logline that doesn't show up when I view in a terminal or text editor. I will continue to investigate this, but in the meantime, any advice here would be greatly appreciated. And, if there is any further information I can provide to help, please let me know. thanks Bryan ---
**Author:** ![bryand](https://avatars.discourse-cdn.com/v4/letter/b/a87d85/32.png) [@bryand](https://discuss.elastic.co/u/bryand)\ **Post date:** [July 23, 2020, 9:21am UTC](https://discuss.elastic.co/t/multiline-issue-with-filebeat/242234/2 "2020-07-23T09:21:41Z")
From testing filebeat with output.stdout enabled I can see that the issue is very much on the filebeat side. The 'multiline.pattern' is not matching the logline. Still investigating why.... ---
**Author:** ![bryand](https://avatars.discourse-cdn.com/v4/letter/b/a87d85/32.png) [@bryand](https://discuss.elastic.co/u/bryand)\ **Post date:** [July 23, 2020, 10:22am UTC](https://discuss.elastic.co/t/multiline-issue-with-filebeat/242234/3 "2020-07-23T10:22:38Z")
Another point that might be worth mentioning is that I am using `filebeat.config.inputs` rather than `filebeat.inputs` ---
**Author:** ![bryand](https://avatars.discourse-cdn.com/v4/letter/b/a87d85/32.png) [@bryand](https://discuss.elastic.co/u/bryand)\ **Post date:** [July 23, 2020, 11:50am UTC](https://discuss.elastic.co/t/multiline-issue-with-filebeat/242234/4 "2020-07-23T11:50:15Z")
> [@bryand](#): > > `/home/filebeat/filebeat/filebeat test config config/filebeat.service1.yml` it turned out that this was relevant. When I sent the output to the console, I could see that the multiline was not working when using `filebeat.config.inputs` when I change it to `filebeat.inputs` I can see that the multiline does work. My problem now is sending that to logstash. It doesn't appear to be working, but thats a different issue. Thanks to anyone who cast an eye over this issue. I guess the solution is not to use `filebeat.config.inputs`. ---
**Author:** ![calanon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/calanon/32/64004_2.png) [@calanon](https://discuss.elastic.co/u/calanon)\ **Post date:** [August 10, 2020, 10:16am UTC](https://discuss.elastic.co/t/multiline-issue-with-filebeat/242234/5 "2020-08-10T10:16:03Z")
You have a valid point about the multiline because I am having exactly the same trouble as you. What worries me though is that there are no replies or comments on this post from other people/experts that could actually help debug this problem. ---
**Author:** ![bryand](https://avatars.discourse-cdn.com/v4/letter/b/a87d85/32.png) [@bryand](https://discuss.elastic.co/u/bryand)\ **Post date:** [August 10, 2020, 11:08am UTC](https://discuss.elastic.co/t/multiline-issue-with-filebeat/242234/6 "2020-08-10T11:08:47Z")
I was adding to this case as I was debugging it, so I guess I arrived at my solution before anyone else may have felt the need to add anything. I found running filebeat in debug mode helped a lot - I used full verbosity. A lot to parse, but it helped me get where I needed to be. [https://www.elastic.co/guide/en/beats/filebeat/current/enable-filebeat-debugging.html](https://www.elastic.co/guide/en/beats/filebeat/current/enable-filebeat-debugging.html) If you want to elaborate on your issues I'd be happy to help, if I can. Or if you have a ticket already opened, link it here and I can share what I learned. thanks ---
**Author:** ![calanon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/calanon/32/64004_2.png) [@calanon](https://discuss.elastic.co/u/calanon)\ **Post date:** [August 10, 2020, 12:29pm UTC](https://discuss.elastic.co/t/multiline-issue-with-filebeat/242234/7 "2020-08-10T12:29:42Z")
Its quite funny because after using your debugging methods I was also able to fix my issue, ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 7, 2020, 2:29pm UTC](https://discuss.elastic.co/t/multiline-issue-with-filebeat/242234/8 "2020-09-07T14:29:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.