I am trying to send the logs with the filebeat mysql module to my logstash and it already obtains the pipeline and does the parsing correctly.
How can I differentiate both inputs ( Slowlog and errror ) to apply different filters to them?
for example:
if [log][file][path] == "/var/lib/mysql/g99dnap797-slow.log" {
elasticsearch {
hosts => ["https://x:9200","https://y:9200","https://z:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-tecnico-mysql-%{[agent][hostname]}-%{+YYYY.MM.dd}"
ssl => true
manage_template => false
user => "elastic"
password => "xxxxxxx"
pipeline => "filebeat-7.13.2-mysql-slowlog-pipeline"
}
}
the module:
- module: mysql
# Error logs
error:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Slow logs
slowlog:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
There are other fields in @metadata that u may be able to use our should be able to use event.dataset which in this case would be mysql.error and mysql.slowlog
In the filter block do the if / else on the event.data type and set a field or tag with part of the index name like index-subtype
then your output could look something like
output {
if [@metadata][pipeline] {
elasticsearch {
hosts => "http://localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{index-subtype}"
pipeline => "%{[@metadata][pipeline]}"
user => "elastic"
password => "secret"
}
} else {
elasticsearch {
hosts => "http://localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{index-subtype}"
user => "elastic"
password => "secret"
}
}
}
Unclear why you want to separate by hostname... that often leads to many many indices with small shard which leads to over sharding... The host.name and agent.hostname is in every event and can be easily filtered ... just a thought
Regarding the name of the indexes, we put the hostname because it is a cluster with quite specific data. Anyway, could you perform a reindex with a date variable so that it just removes the hostname?
That is not going to work that destination index will be the literal of what you defined... and of course timestamp go down to the ms so you would probably not want to do that...
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.