I finally got the filebeat error module for apache to parse my error logs. It was a combination of looking through the
GET _ingest/processor/grok
In dev tools, and fairly heavily modifying the apache error module configuration. We did add a custom field to our logs that adds the domain name to the front of the apache access and error logs - but the error logs weren't parsing correctly before that change anyway, but apache logs were working all along.
Here is what we currently have in the module to make it work:
{
"description": "Pipeline for parsing apache error logs",
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"(%{DATA:server.name} : )?\\[%{HTTPDERROR_DATE:apache.error.timestamp}\\] \\[(:)?%{LOGLEVEL:log.level}\\]( \\[client %{IPORHOST:source.address}\\])? %{GREEDYDATA:message}",
"(%{DATA:server.name} : )?\\[%{HTTPDERROR_DATE:apache.error.timestamp}\\] \\[(:)?%{LOGLEVEL:log.level}\\] \\[pid %{NUMBER:process.pid:long}(:tid %{NUMBER:process.thread.id:long})?\\]( \\[client %{IPORHOST:source.address}\\])? %{GREEDYDATA:message}"
],
"ignore_missing": true
}
},
{
"remove": {
"field": "apache.error.timestamp",
"ignore_failure": true
}
},
{
"grok": {
"field": "source.address",
"ignore_missing": true,
"patterns": [
"^(%{IP:source.ip}|%{HOSTNAME:source.domain})$"
]
}
},
{
"geoip": {
"field": "source.ip",
"target_field": "source.geo",
"ignore_missing": true
}
}
],
"on_failure" : [{
"set" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
}]
}
Notice to make it work we had to use
%{HTTPDERROR_DATE}
Instead of
%{APACHE_TIME}
However, it refused to rake in any of the error logs until we also removed the following two sections from the file
"pattern_definitions": {
"APACHE_TIME": "%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}"
},
AND
{
"date": {
"field": "apache.error.timestamp",
"target_field": "@timestamp",
"formats": ["EEE MMM dd H:m:s yyyy", "EEE MMM dd H:m:s.SSSSSS yyyy"],
"ignore_failure": true
}
Of course for this to work we had to
DELETE _ingest/pipeline/filebeat-7.0.0-apache-error-pipeline
Which was added back in after restarting filebeat
Still have some polishing up to do on the grok patterns to get everything I want, which is easy enough, but wanted to bring this to attention.
I think our logs are maybe not normally formatted apache error logs, however, I am fairly certain that either way the following is broken and no longer works
%{APACHE_TIME}
Thoughts?