I'm a little bit struggling with Logtash parsing mysql-slow.log file.
I configured Filebeat to send logs from the file into Elasticsearch through Logstash so I can see recieved messages in Kibana like this:
Nov 5, 2019 @ 12:14:26.250 /var/log/mysql/mysql-slow.log
# User@Host: user[user] @ localhost [] Id: 2
# Query_time: 0.000572 Lock_time: 0.000137 Rows_sent: 2 Rows_examined: 2
SET timestamp=1572952461;
select * from Persons;
I'd like to parse the message by using Logstash but every pattern I used led to some problem in communication between Filebeat and Logstash. For example I used the following configuration file for Logstash:
And it doesn't work because multiline codec isn't supported with beats... I set multiline in input section of filebeat.yml file:
multiline.pattern: '^#'
multiline.negate: true
multiline.match: after
So now I can run Logstash and Filebeat but filters don't match with lines and it doesn't give me any fields that are defined in config file in Logstash.
I thought that when I change multiline input in filebeat.yml file it sends me the myslq-slow.log in one line format.
My filebeat.yml:
# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
multiline.pattern: '^#'
# Defines if the pattern set under pattern should be negated or not. Default is false.
multiline.negate: true
# Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
# that was (not) matched before or after or as long as a pattern is not matched based on negate.
# Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
multiline.match: after
And I still get grokparsefailure.
How to achieve that logstash recieves the following message:
Thanks for reply! I found out that there is not a problem with filebeat but with Logstash. I thought that logstash can parse only one line messages so I was trying to merge several lines into one via filebeat so logstash could recieve it as one line and parse it into fields I need but there is an option to parse it as it is so mysql-slow.log messages
Thanks I didn't get the meaning of filebeat multiline input. I thought it would make one line from several lines but it just gives rule into the struture and informs logstash that there is multiline event coming?
I do not think I can explain multiline handling in filebeat any better than the documentation does. It combines multiple lines into the message field a single event (separated by newline characters).
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.