Not parsing apache access and error logs

Elastic Stack Logstash
1

I am using filebeat to send apache logs to logstash on my elastic server. I can see the log entries with event.dataset set to apache.access , but they aren't being parsed, so no info in dashboard apache screens. Also, when Iook at Observability...logs on the demo system it has the [apache][access] prefix per line and when I "view details" it shows fields like http.request.method.

Both machines Centos 7

ELK machine
elasticsearch.x86_64               7.9.0-1
filebeat.x86_64                    7.9.0-1
kibana.x86_64                      7.9.0-1
logstash.noarch                    1:7.9.0-1
metricbeat.x86_64                  7.9.0-1

Apache machine
filebeat.x86_64                      7.9.0-1
metricbeat.x86_64                    7.9.0-1

In my apache host file, I have made sure the format as required.

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
CustomLog /var/log/httpd/mysystem-access.log combined

/etc/filebeat/modules.d/apache.yml on the web server:

# Module: apache
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.9/filebeat-module-apache.html

- module: apache
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/httpd/mysystem-access.log"]

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/httpd/mysystem-error.log"]

On ELK machine,
ll /etc/logstash/conf.d/

-rw-r--r--. 1 root root  330 Sep  1 21:06 02-beats-input.conf
-rw-r--r--. 1 root root 1426 Jan 28 16:07 10-apache-filter.conf
-rw-r--r--. 1 root root 3148 Jul 31 18:08 10-syslog-filter.conf
-rw-r--r--. 1 root root  169 Sep  4 23:13 30-elasticsearch-output.conf

/etc/logstash/conf.d/10-apache-filter.conf

        filter {
           if [type] in [ "apache" , "apache_access" , "apache-access" ]  {
              grok {
                 match => [
                 "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}",
                 "message" , "%{COMMONAPACHELOG}+%{GREEDYDATA:extra_fields}"
                 ]
                 overwrite => [ "message" ]
              }
              mutate {
                 convert => ["response", "integer"]
                 convert => ["bytes", "integer"]
                 convert => ["responsetime", "float"]
              }
              geoip {
                 source => "clientip"
                 target => "geoip"
                 add_tag => [ "apache-geoip" ]
              }
              date {
                 match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
                 remove_field => [ "timestamp" ]
              }
              useragent {
                 source => "agent"
              }
           }
           if [type] in ["apache_error","apache-error"] {
              grok {
                 match => ["message", "\[%{WORD:dayname} %{WORD:month} %{DATA:day} %{DATA:hour}:%{DATA:minute}:%{DATA:second} %{YEAR:year}\] \[%{NOTSPACE:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:message}"]
                 overwrite => [ "message" ]
              }
              mutate
              {
                 add_field =>
                 {
                    "time_stamp" => "%{day}/%{month}/%{year}:%{hour}:%{minute}:%{second}"
                 }
              }
              date {
                 match => ["time_stamp", "dd/MMM/YYYY:HH:mm:ss"]
                 remove_field => [ "time_stamp","day","dayname","month","hour","minute","second","year"]
              }
           }
        }

When I go to the Grok debugger and have

Sample data taken from the log shown on Observability...Logs

127.0.0.1 - - [28/Jan/2021:16:05:49 +0000] "GET /server-status?auto= HTTP/1.1" 200 482 "-" "Go-http-client/1.1

Grok pattern

%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}

I get the correctly parsed

{
  "request": "/server-status?auto=",
  "agent": "\"Go-http-client/1.1\"",
  "auth": "-",
  "ident": "-",
  "verb": "GET",
  "referrer": "\"-\"",
  "response": "200",
  "bytes": "482",
  "clientip": "127.0.0.1",
  "httpversion": "1.1",
  "extra_fields": "",
  "timestamp": "28/Jan/2021:16:05:49 +0000"
}

So it would appear that the filter isn't firing.

2

Apologies.. Missread the question.

Perhaps I am a little confused... if you are using one of the supported formats you should not need to parse, or are you intentionally parsing this your self.

Question for a test did you run filebeat setup -e this is crucial to make sure the correct ingest pipelines were loaded.

I often test sending from Filebeat directly to Elasticsearch first before passing trough logstash as a test.

Did you try to just sending directly from filebeat to Elasticsearch as a test and see if you are getting correct parsing?

If so then you don't need to do any parsing in logstash you just use it as a pass through..

It could look like this... this treats logstash as a pass through and then will use the automatically configured ingest pipelines to parse the apache logs... if they are one of the supported formats.

################################################
# beats->logstash->es default config.
################################################
input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      user => "elastic"
      password => "secret"
    }
  }
}

Also as a note when you set up your conf.d that way think of all the .conf files being concatenated... which I think you get.

3

Thanks for your reply. I was definitely confused.

This started all started off at version 7.8 some time ago, Then I accidentally upgraded some items to 7.9 so I updated everything to 7.9, so possibly things are a bit half -baked.

In the end, I did the filebeat setup -e on the ELK machine as I couldn't get the permissions sorted to do it from my apache machine. this added the apache ingest pipeline which was previously missing.

I then re-did my logstash modules.d directory to reflect the outputs of your example and removed my apache filter from there.

It now works ok.

Many thanks.

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.