Multiline issue with filebeat

Elastic Stack Beats
1

Hi,

I'm having issues with trying to aggregate java exceptions into a single message using the filebeat multiline support. From everything I've read this should be very simple, yet, no matter what I try, it doesn't work. I've spent a few hours reading through similar issues that people have had, but none of the suggestions made for those, have worked for me.

My setup is;
filebeat -> logstash -> elasticsearch

all logstash is doing in this setup is directing the message to a particular elasticsearch index depending on the values of some metadata tags that are added by filebeat.

I should add that we are running multiple services and each have their own instance of filebeat running - we are running on ubuntu 16.04, and filebeat is controlled by systemd.

For each of the services, the multiline config is;

  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

where the log line begins with the date. An example of the logline is;

2020-07-22 15:41:24,174 - [WARN] - o.a.k.c.n.Selector - Error in I/O with hostA java.io.EOFException: null
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62) ~[service1.jar:1.0]
	at org.apache.kafka.common.network.Selector.poll(Selector.java:248) ~[service1.jar:1.0]
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) [service1.jar:1.0]
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) [service1.jar:1.0]
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) [service1.jar:1.0]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_211]

I have run this regexp through the play sandbox (https://play.golang.org/) with the above logline, and it matches fine. I have even tried variations on the regexp, as follows

  #multiline.pattern: '^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]'
  #multiline.pattern: '^\d?\d.\d\d.\d\d'

and while both return matches in the sandbox, I see no difference in the behaviour when I try them with my service.

I have also tested my config with filebeat, as follows;

root@<host>:/# /home/filebeat/filebeat/filebeat test config config/filebeat.service1.yml
Config OK

and I have used a yaml validator to ensure there are no formatting issues with my config file.

At this point I am not sure what else I can do to pinpoint what my issue is.

Firstly, can I discount logstash as being an issue here? The logstash config for this service does very little. It simply takes in the message and outputs it to a certain index. The logstash config is

input {
  beats {
    port => 5044
  }
}

filter {
  if "service1" in [tags]{
    grok {
      match => {"message" => "INFO\s+\[%{TIMESTAMP_ISO8601:logtimestamp}\] %{GREEDYDATA:message}"}
    }
  }
}
output {
  if "service1" in [tags]{
    elasticsearch {
      hosts => ["http://<elasticsearch_host>:9200"]
      index => "<servicename>-%{+YYYY.MM.dd}"
    }
  }
}

My own feeling is that logstash is not an issue and that I need to focus on the filebeat side. Unfortunately the logs are not terribly verbose and there is nothing to indicate whether my logline is being aggregated into a single message, or not.

One thing that did occur to me is that there could be some hidden character in the logline that doesn't show up when I view in a terminal or text editor. I will continue to investigate this, but in the meantime, any advice here would be greatly appreciated. And, if there is any further information I can provide to help, please let me know.

thanks
Bryan

2

From testing filebeat with output.stdout enabled I can see that the issue is very much on the filebeat side. The 'multiline.pattern' is not matching the logline. Still investigating why....

3

Another point that might be worth mentioning is that I am using

filebeat.config.inputs

rather than

filebeat.inputs

4

it turned out that this was relevant.

When I sent the output to the console, I could see that the multiline was not working when using
filebeat.config.inputs

when I change it to
filebeat.inputs

I can see that the multiline does work.

My problem now is sending that to logstash. It doesn't appear to be working, but thats a different issue. Thanks to anyone who cast an eye over this issue. I guess the solution is not to use
filebeat.config.inputs.

5

You have a valid point about the multiline because I am having exactly the same trouble as you.

What worries me though is that there are no replies or comments on this post from other people/experts that could actually help debug this problem.

6

I was adding to this case as I was debugging it, so I guess I arrived at my solution before anyone else may have felt the need to add anything.

I found running filebeat in debug mode helped a lot - I used full verbosity. A lot to parse, but it helped me get where I needed to be.

https://www.elastic.co/guide/en/beats/filebeat/current/enable-filebeat-debugging.html

If you want to elaborate on your issues I'd be happy to help, if I can. Or if you have a ticket already opened, link it here and I can share what I learned.

thanks

7

Its quite funny because after using your debugging methods I was also able to fix my issue,

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.