Apache Access logs not appearing in Elasticsearch/Kibana

Elastic Stack Logstash
1

Hi,

We’ve been successfully ingesting our Apache logs however as of around 2 weeks ago the access logs have stopped appearing however the error logs are still showing up in ES/Kibana. We did upgrade from 8.19.2 to 8.19.3 around that time but the timing doesn’t 100% correlate.

There are no error logs appearing in either logstash or ES and if I create an output file filter in logstash I see entries for the Apache access logs in there:

{"message":"10.65.193.100 - - [19/Sep/2025:14:44:13 +0100] \"GET /status/api/server_logs/status HTTP/1.1\" 401 802 \"https://server.foo/status/app/server_logs\" \"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:142.0) Gecko/20100101 Firefox/142.0\"","user_agent":{"name":"Firefox","device":{"name":"{\"name\":\"Other\"}"},"original":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:142.0) Gecko/20100101 Firefox/142.0","os":{"name":"%{[ua_tmp][os_name]}"}},"@version":"1","host":{"name":"node.foo"},"apache":{"access":{}},"agent":{"hostname":"node.foo","version":"7.17.22","ephemeral_id":"ba05901f-49f8-4c38-91bd-f2f44acec47e","name":"node.foo","type":"filebeat","id":"8cc85ed2-6fec-4c43-833a-0c51d16ae3dc"},"source":{"address":"10.65.193.100","ip":["10.65.193.100","10.65.193.100"],"as":{},"geo":{}},"http":{"response":{"body":{"bytes":"802"},"status_code":"401"},"request":{"method":"get","referrer":"https://server.foo/status/app/server_logs"},"version":"1.1"},"labels":{"tier":"test"},"url":{"original":"/status/api/server_logs/status"},"event":{"outcome":"failure","created":"2025-09-19T13:44:13.046Z","category":"web","kind":"event","original":"10.65.193.100 - - [19/Sep/2025:14:44:13 +0100] \"GET /status/api/server_logs/status HTTP/1.1\" 401 802 \"https://server.foo/status/app/server_logs\" \"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:142.0) Gecko/20100101 Firefox/142.0\"","module":"apache","dataset":"apache.access","type":"access"},"ecs":{"version":"1.12.0"},"fileset":{"name":"access"},"service":{"name":"service","type":"class"},"log":{"file":{"path":"/var/log/apache2/moo_access_log"},"offset":23417700},"input":{"type":"log"},"tags":["beats","beats_input_codec_plain_applied","_dateparsefailure","_geoip_lookup_failure"]}

(We were seeing dateparsefailures previously and in the successfully-stored error logs.

We’re using the following grok rules to capture + tokenise our Apache access logs (this hasn’t changed).

      grok {
        match => {
          "message" => [
            "%{IPORHOST:[destination][domain]} %{IPORHOST:[source][ip]} - (-|%{D
ATA:[user][name]}) \[%{HTTPDATE:[apache][access][time]}\] \"(?:%{WORD:[http][req
uest][method]} %{DATA:[url][original]} HTTP/%{NUMBER:[http][version}|-)?\" %{NUM
BER:[http][response][status_code]:long} (?:%{NUMBER:[http][response][body][bytes
]:long}|-)( \"%{DATA:[http][request][referrer]}\")?( \"%{DATA:[user_agent][origi
nal]}\")?",
            "%{IPORHOST:[source][address]} - (-|%{DATA:[user][name]}) \[%{HTTPDA
TE:[apache][access][time]}\] \"(?:%{WORD:[http][request][method]} %{DATA:[url][o
riginal]} HTTP/%{NUMBER:[http][version]}|-)?\" %{NUMBER:[http][response][status_
code]:long} (?:%{NUMBER:[http][response][body][bytes]:long}|-)( \"%{DATA:[http][
request][referrer]}\")?( \"%{DATA:[user_agent][original]}\")?",
            "%{IPORHOST:[source][address]} - (-|%{DATA:[user][name]}) \[%{HTTPDA
TE:[apache][access][time]}\] \"-\" %{NUMBER:[http][response][status_code]:long} 
-",
            "\[%{HTTPDATE:[apache][access][time]}\] %{IPORHOST:[source][address]
} %{DATA:[apache][access][ssl][protocol]} %{DATA:[apache][access][ssl][cipher]} 
\"%{WORD:[http][request][method]} %{DATA:[url][original]} HTTP/%{NUMBER:[http][v
ersion]}\" (-|%{NUMBER:[http][response][body][bytes]:long})"
          ]
        }
        tag_on_failure => "_apache_access_grokparsefailure"
      }

Given the lack of error output and that the data appears to be getting sent via Logstash we’re at a bit of a loss as to what’s causing the issue or where to look to diagnose it. I’ve had a look at the index templates but all the fields appear to be present and correct.

2

Hi, welcome!

Your access logs are reaching Logstash but likely fail at grok/date parsing.

  • Check if events have _apache_access_grok_failure tag.

  • Temporarily disable the date filter (possible dateparsefailure).

  • Add stdout { codec => rubydebug } to confirm logs are parsed.

  • If raw logs reach ES without filters, issue is grok/date mapping

This will show where they’re being dropped.

Elastic Stack > Logstash

1 Like
3

You should use the apache OOB grok pattern:
%{HTTPD_COMBINEDLOG}

4

Hi,

Thanks, a few useful pointers in there for sure (codec => rubydebug makes the output file so much easier to read)! I don’t see any events (ever) with the ‘_apache_access_grok_failure’ tag. I’ve commented out the date post-processing and am simply doing the following pattern:

\[%{HTTPDATE:[timestamp]}\]

I’m now getting the access logs showing up in Kibana again without a dateparsefailure error:

     "timestamp" => "22/Sep/2025:10:58:08 +0100",
    "@timestamp" => 2025-09-22T09:58:09.296Z,

(Duplicate field mapping but can live with this for now).

I suppose now the questions are: 1. why has this grok code broken all of a sudden when it’s been working since we were on ES 6?) 2. Why are the Apache error logs still being populated in Elastic despite also throwing a dateparsefailure error? 3. Why is Logstash failing to parse the following input “[22/Sep/2025:10:58:08 +0100]” with the grok pattern (and why isn’t it throwing any errors in the logs :

      mutate { rename => { "@timestamp" => "[event][created]" } }
      date {
        match => [
          "[apache][access][time]",
          "dd/MMM/yyyy:H:m:s Z"
        ]
        target => "@timestamp"
      }
      mutate { remove_field => [ "[apache][access][time]" ] }

Looking at the date field docs at Date filter plugin | Logstash Reference [8.19] | Elastic the pattern looks correct to me (though it also looks like I could/should be using the double-digit HH:mm:ss format for date values like ‘[21/Sep/2025:03:27:44 +0100]’?

5

The issue is with the date filter pattern.

Your Apache log timestamp is like:

[22/Sep/2025:10:58:08 +0100]

But in your config you used:

"dd/MMM/yyyy:H:m:s Z"

Here’s the problem:
• H:m:s → uses single-digit minute/second.
• Your logs have two-digit minutes/seconds (10:58:08).

Correct pattern:

date {
match => [ "[apache][access][time]", "dd/MMM/yyyy:HH:mm:ss Z" ]
target => "@timestamp"
}

That should stop the dateparsefailure and avoid the duplicate field mapping.

6

Hi, I’ve corrected the date parting line but I’m still getting a dateparsefailure error. What’s interesting is that there is no "@timestamp” in the logstash output at all:

date {
        match => [
          "[apache][access][time]",
          "dd/MMM/yyyy:HH:mm:ss Z"
        ]
        target => "@timestamp"
      }

Having done a bit more playing around it appears to be that if the “@timestamp” field is absent then the log entry doesn’t appear in ES/Kibana.

I also wondered if there was some issues with mismatches between data field types e.g. storing the placeholder timestamp from the log file as a date rather than a string.

I’ve got the current and logs are now appearing in ES/Kibana with the timestamp set to the default ingestion timestamp however the event.created timestamp I’m now trying to populate is still absent and there’s still a dateparse error in the logs:

      mutate { copy => { "@timestamp" => "[event][ingested]" } }
      date {
        match => [
          "[apache][access][time]",
          "dd/MMM/yyyy:HH:mm:ss Z"
        ]
        #target => "@timestamp"
        target =>"[event][created]"
      }
      mutate { remove_field => [ "[apache][access][time]" ] }

So overall at least now we’re not losing logs but it would be nice if the grok parsing of the date from the logfile itself worked.

7

When an event is created the [@timestamp] field is always added. If you have events that do not have that field the only explanation I can think of is that you removed it. The fix would be to not do that.

If I recall correctly then the @timestamp field is required in elasticsearch, so unless you are using an ingest pipeline to add it to documents that don’t have one I would expect (based on earlier versions) to get an exception in the logstash logs for events like that.

1 Like
8

Yep that makes sense. If the mutate → rename step was renaming an empty field to be the @timestamp field then that would break things. Still a wee bit of a mystery as to why it stopped working (the only thing that changed on the clusters was rolling over from 8.19.2 to 8.19.3) and the date parsing still isn’t quite working but at least I’ve now got logs going in (and a good example of why copying rather than moving/renaming is safer!). I notice that 8.19.4 is now out so will try upgrading to that shortly.