Hi,
I'm having issues with trying to aggregate java exceptions into a single message using the filebeat multiline support. From everything I've read this should be very simple, yet, no matter what I try, it doesn't work. I've spent a few hours reading through similar issues that people have had, but none of the suggestions made for those, have worked for me.
My setup is;
filebeat -> logstash -> elasticsearch
all logstash is doing in this setup is directing the message to a particular elasticsearch index depending on the values of some metadata tags that are added by filebeat.
I should add that we are running multiple services and each have their own instance of filebeat running - we are running on ubuntu 16.04, and filebeat is controlled by systemd.
For each of the services, the multiline config is;
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after
where the log line begins with the date. An example of the logline is;
2020-07-22 15:41:24,174 - [WARN] - o.a.k.c.n.Selector - Error in I/O with hostA java.io.EOFException: null
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62) ~[service1.jar:1.0]
at org.apache.kafka.common.network.Selector.poll(Selector.java:248) ~[service1.jar:1.0]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) [service1.jar:1.0]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) [service1.jar:1.0]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) [service1.jar:1.0]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_211]
I have run this regexp through the play sandbox (https://play.golang.org/) with the above logline, and it matches fine. I have even tried variations on the regexp, as follows
#multiline.pattern: '^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]'
#multiline.pattern: '^\d?\d.\d\d.\d\d'
and while both return matches in the sandbox, I see no difference in the behaviour when I try them with my service.
I have also tested my config with filebeat, as follows;
root@<host>:/# /home/filebeat/filebeat/filebeat test config config/filebeat.service1.yml
Config OK
and I have used a yaml validator to ensure there are no formatting issues with my config file.
At this point I am not sure what else I can do to pinpoint what my issue is.
Firstly, can I discount logstash as being an issue here? The logstash config for this service does very little. It simply takes in the message and outputs it to a certain index. The logstash config is
input {
beats {
port => 5044
}
}
filter {
if "service1" in [tags]{
grok {
match => {"message" => "INFO\s+\[%{TIMESTAMP_ISO8601:logtimestamp}\] %{GREEDYDATA:message}"}
}
}
}
output {
if "service1" in [tags]{
elasticsearch {
hosts => ["http://<elasticsearch_host>:9200"]
index => "<servicename>-%{+YYYY.MM.dd}"
}
}
}
My own feeling is that logstash is not an issue and that I need to focus on the filebeat side. Unfortunately the logs are not terribly verbose and there is nothing to indicate whether my logline is being aggregated into a single message, or not.
One thing that did occur to me is that there could be some hidden character in the logline that doesn't show up when I view in a terminal or text editor. I will continue to investigate this, but in the meantime, any advice here would be greatly appreciated. And, if there is any further information I can provide to help, please let me know.
thanks
Bryan