Hi. I just want to preserve the original file name for example: laravel-2022-05-17.log
and set it as index.
I was not able to find the propper way to obtain it using filebeat and sending it to logstash.
Thanks
Hi. I just want to preserve the original file name for example: laravel-2022-05-17.log
and set it as index.
I was not able to find the propper way to obtain it using filebeat and sending it to logstash.
Thanks
Hi, solved. For each file I wanted to have a custom index name I had to define a different log. My two log files are userimports.log
and push-notifications.log
Filebeat.yml:
filebeat.inputs:
# Laravel Logs
- type: log
enabled: true
paths:
- /var/log/dmesg_log/userimports/push-notifications.log
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
fields:
logType: "laravel"
tags: ["push-notifications"]
- type: log
enabled: true
paths:
- /var/log/dmesg_log/userimports/userimports.log
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
fields:
logType: "laravel"
tags: ["userimports"]
# Enrich laravel logs with docker data (not working)
# processors:
# - add_docker_metadata:
# host: "unix:///var/run/docker.sock"
# setup.kibana:
# host: "http://elasticsearch:5601"
output.logstash:
hosts: ["logstash:5044"]
And now using the tags we can determine which log is which in logstash.conf:
input {
beats {
port => 5044
}
}
##filter data filtering operation
filter {
grok {
match => {
"message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:env}\.%{DATA:severity}: (?<log>[^{]+)?%{GREEDYDATA:raw-json}"
}
}
grok {
match => {
"raw-json" => "(?<raw-process-json>\{(.*)\})%{GREEDYDATA:response}"
tag_on_failure => [ ]
}
}
json {
source => "raw-process-json"
target => "json"
}
mutate {
rename => { "message" => "raw-message" }
rename => { "json" => "raw-process-json" }
}
}
output {
if "push-notifications" in [tags] {
elasticsearch{
hosts => ["http://elasticsearch:9200"]
index => "push-notifications-%{+YYYY.MM.dd}"
}
stdout {}
}
if "userimports" in [tags] {
elasticsearch{
hosts => ["http://elasticsearch:9200"]
index => "userimports-%{+YYYY.MM.dd}"
}
stdout {}
}
}
Hope it helps!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.