Hello all,
I'm trying use ELK with Filebeat to parse some apache logs, but the filter never gets fired.
ELK Server:
Packages installed:
logstash-5.0.2-1.noarch
kibana-5.0.2-1.x86_64
elasticsearch-5.0.2-1.noarch
The filter is OK:
0:> /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/apache-filter.conf --config.test_and_exit --path.settings /etc/logstash/
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK
Content of /etc/logstash/conf.d/apache-filter.conf:
filter {
if [type] == "apache-access" {
# This will parse the apache access event
grok {
match => [ "message", "%{COMBINEDAPACHELOG}" ]
}
}
}
Client server(filebeat):
Packages installed:
filebeat-5.0.2-1.x86_64
Contents of /etc/filebeat/filebeat.yml
`filebeat.prospectors:
- input_type: log
paths:- /logs/*.log
document_type: apache-access
output.elasticsearch:
hosts: ["mylogstashserver.com:9200"]
tls:
certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]`
- /logs/*.log
Filebeat is working and the log is forwarded to Logstash and I can see on Kibana, below an example:
{
"_index": "filebeat-2016.12.07",
"_type": "apache-access",
"_id": "AVjZG5H0oxxqAxp3g61g",
"_score": null,
"_source": {
"@timestamp": "2016-12-07T11:46:51.405Z",
"beat": {
"hostname": "myfilebeatserver.com",
"name": "myfilebeatserver.com",
"version": "5.0.2"
},
"input_type": "log",
"message": "191.255.248.250 - - [05/Dec/2016:11:21:05 +0000] "GET http://img.mysite.com/8019370/orig-img.mysite.com/cars/389632.jpg HTTP/1.1" 200 14383 "http://www.mysite.com/travel/resultado-de-carros?searchtype=Car&withdrawalCity=FLL&withdrawalCityType=AEROPORTO&devolutionCity=FLL&devolutionCityType=AEROPORTO&anotherCity=false&pickUpDateTime=20/12/2016T14:00&returnDateTime=04/01/2017T14:00&pickUpLocationCityCode=34691&returnLocationCityCode=34691\" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36" "-"",
"offset": 283053696,
"source": "/logs/site/wpc_19370_20161205_0001.log",
"type": "apache-access"
},
"fields": {
"@timestamp": [
1481111211405
]
},
"sort": [
1481111211405
]
}
I tried the example here Logstash configuration examples | Logstash Reference [8.11] | Elastic, and the content of the message is parsed correctly. Looks like the issue is when the content comes from Filebeat, I already configured Logstash log to Debug and got nothing. Any idea?
Best Regards,
Danilo F. Chilene