Basic Apache Filebeat Log question - missing fields

Elastic Stack Beats
1

Hi,
Sorry for the newbie question: I'm able to pull in my apache logs and when I try to filter by apache2.access.response_code , I get zilch.

I looked and the response code data is in the message from apache.

Is the problem that I'm having because I'm going directly from the filebeat to elastic? And that I need to use logstash in the middle to expose the apache2.access.response_code parameter? Just want to make sure I'm on the right path.

Thanks!

2

Hi @Shorthills, welcome to the Elastic community forums!

You should be able to go directly from Filebeat to Elasticsearch for what you're trying to do — ingest Apache logs. Would you mind sharing a few pieces of information to help figure out what's going on?

  1. The output of filebeat version

  2. The output of filebeat modules list

  3. Your complete filebeat.yml file (but please mask out any sensitive information in it before posting).

  4. Any errors or warnings in your Filebeat log after starting it up.

  5. The output of GET _cat/indices/filebeat*, run against your Elasticsearch cluster.

  6. The output of GET _cat/templates/filebeat*, run against your Elasticsearch cluster.

Thanks,

Shaunak

3

  1. ilebeat version 7.5.1 (amd64), libbeat 7.5.1 [60dd883ca29e1fdd5b8b075bd5f3698948b1d44d built 2019-12-16 21:56:14 +0000 UTC]

  2. Enabled:
    apache

Disabled:
apache2
auditd
aws
azure
cef
cisco
coredns
elasticsearch
envoyproxy
googlecloud
haproxy
ibmmq
icinga
iis
iptables
kafka
kibana
logstash
misp
mongodb
mssql
mysql
nats
netflow
nginx
osquery
panw
postgresql
rabbitmq
redis
santa
suricata
system
traefik
zeek

I'm working on getting you the rest :slight_smile:

Thanks!

4

Hi @Shorthills, thanks for reporting the Filebeat version. You will find the response code in http.response.status_code. In general, you can find the fields exported by the Filebeat apache module here: https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-apache.html.

Shaunak

5

Thank you. I'm not seeing it on the list of my available fields. Does that matter? The issue is that none of these exported fields have data, unless I'm looking in the wrong place. I figured that I'd be able to see the data in the Discover Module. And the logs are there, they're just coming in as Message and not as parsed data (which is why I asked about logstash).

As for my filebeat.yml, it's default except for this change:

  1. I didn't want to pull all of the system logs so I pulled apache only.
  2. Also, I added the php error logs b/c I do need that reporting
    # /var/log/*.log
    - /var/log/apache2/error_log
    #- c:\programdata\elasticsearch\logs\*
    - /home/*/logs/*.php.error.log

Will the last ```/home/*/...` line affect the apache log parsing? I didn't edit the apache.yml file at all.

And I'm getting the rest of the data.

6

And is this the info re: indexes and index templates? I think that I created too many indexes b/c every time I adjusted the .yml config file I regenerated the index which I thought was what I had to do (but now I'm thinking that I didn't need to do that). Thanks for your continued help.

2019-12-26_21-02-20
2019-12-26_21-02-20873×786 149 KB

2019-12-26_21-02-49
2019-12-26_21-02-49895×906 135 KB

7

And if it helps, the filebeat 6.8.6 index template has the following:

        "apache2.access.ssl.protocol",
        "apache2.access.ssl.cipher",
        "apache2.access.user_name",
        "apache2.access.method",
        "apache2.access.url",
        "apache2.access.http_version",
        "apache2.access.referrer",
        "apache2.access.agent",
        "apache2.access.user_agent.device",
        "apache2.access.user_agent.patch",
        "apache2.access.user_agent.name",
        "apache2.access.user_agent.os",
        "apache2.access.user_agent.os_name",
        "apache2.access.geoip.continent_name",
        "apache2.access.geoip.country_iso_code",
        "apache2.access.geoip.region_name",
        "apache2.access.geoip.city_name",
        "apache2.access.geoip.region_iso_code",
        "apache2.error.level",
        "apache2.error.client",
        "apache2.error.message",
        "apache2.error.module",

but nothing with the http. prefix.

===

And these are the only ones listed in the filebeat-7.5.1 template:

        "http.request.body.content",
        "http.request.method",
        "http.request.referrer",
        "http.response.body.content",
        "http.version",

        "apache.access.ssl.cipher",
        "apache.error.module",

====
Should I be using the Apache2 or the Apache module? I was reading and it seems that I should use the Apache one?

Thx

8

And on this box I have

apache.yml and apache2.yml.disabled if that helps.

9

Can you share your compete filebeat.yml and your complete modules.d/apache.yml please?

Also, if possible, could you please share a couple of raw lines from your Apache error log (the one you want to parse)?

Thanks,

Shaunak

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.