When tried to use default COMBINEDAPACHELOG for NGINX access logs, all fields are getting parsed correctly except agent field which contains useragent information. (https://www.elastic.co/guide/en/logstash/current/config-examples.html)
I have used filebeat to send events to logstash.
So logstash by default generates filebeat information in agent field.
It sees duplicate field name and ignores useragent information from NGINX access logs.
I used mutate filter to rename filebeat agent field to beatagent and then I was able to collect agent information correctly from access logs.
I am new to Logstash and wanted to know doing something like this is okay and shouldnt logstash have capability to differentiate between agent information from access logs and filebeat agent?
input {
beats {
port => 5044
}
}
filter {
mutate {
rename => {"agent" => "beatagent"}
}
grok {
match => [ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}"]
overwrite => [ "message" ]
}
date {
match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
remove_field => [ "timestamp" ]
}
}
output {
file {
path => "/var/log/output.txt"
}
}