Filebeat 7 Apache Module Error Events Broken

Elastic Stack Beats
1

I finally got the filebeat error module for apache to parse my error logs. It was a combination of looking through the

GET _ingest/processor/grok

In dev tools, and fairly heavily modifying the apache error module configuration. We did add a custom field to our logs that adds the domain name to the front of the apache access and error logs - but the error logs weren't parsing correctly before that change anyway, but apache logs were working all along.

Here is what we currently have in the module to make it work:

{
  "description": "Pipeline for parsing apache error logs",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "(%{DATA:server.name} : )?\\[%{HTTPDERROR_DATE:apache.error.timestamp}\\] \\[(:)?%{LOGLEVEL:log.level}\\]( \\[client %{IPORHOST:source.address}\\])? %{GREEDYDATA:message}",
          "(%{DATA:server.name} : )?\\[%{HTTPDERROR_DATE:apache.error.timestamp}\\] \\[(:)?%{LOGLEVEL:log.level}\\] \\[pid %{NUMBER:process.pid:long}(:tid %{NUMBER:process.thread.id:long})?\\]( \\[client %{IPORHOST:source.address}\\])? %{GREEDYDATA:message}"
        ],
        "ignore_missing": true
      }
    },
    {
      "remove": {
        "field": "apache.error.timestamp",
        "ignore_failure": true
      }
    },

    {
        "grok": {
            "field": "source.address",
            "ignore_missing": true,
            "patterns": [
                "^(%{IP:source.ip}|%{HOSTNAME:source.domain})$"
            ]
        }
    },
    {
        "geoip": {
            "field": "source.ip",
            "target_field": "source.geo",
            "ignore_missing": true
        }
    }
  ],
  "on_failure" : [{
    "set" : {
      "field" : "error.message",
      "value" : "{{ _ingest.on_failure_message }}"
    }
  }]
}

Notice to make it work we had to use

%{HTTPDERROR_DATE}

Instead of

%{APACHE_TIME}

However, it refused to rake in any of the error logs until we also removed the following two sections from the file

        "pattern_definitions": {
          "APACHE_TIME": "%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}"
        },

AND

      {
      "date": {
        "field": "apache.error.timestamp",
        "target_field": "@timestamp",
        "formats": ["EEE MMM dd H:m:s yyyy", "EEE MMM dd H:m:s.SSSSSS yyyy"],
        "ignore_failure": true
      }

Of course for this to work we had to

DELETE _ingest/pipeline/filebeat-7.0.0-apache-error-pipeline

Which was added back in after restarting filebeat

Still have some polishing up to do on the grok patterns to get everything I want, which is easy enough, but wanted to bring this to attention.

I think our logs are maybe not normally formatted apache error logs, however, I am fairly certain that either way the following is broken and no longer works

%{APACHE_TIME}

Thoughts?

2

Couple example logs:

sub.domain.com : [Wed Apr 17 13:25:02 2019] [error] [pid 22208] sapi_apache2.c(325): [client 123.45.67.89:57885] PHP Notice:  Undefined variable: emailadmin in /var/www/html/sub.domain.com/docs/includes/functions.php on line 254

sub.domain.com : [Wed Apr 17 15:02:04.727578 2019] [:error] [pid 12456:tid 47858466629376] [client 123.45.67.89:59030] File does not exist: /var/www/html/index.php
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.