Filebeat apache module not collecting any logs

Elastic Stack Beats
1

So I have been playing around with collecting data using filebeat and sending it via sidecar to my graylog server. I wanted to try out the apache module, so I wrote the configuration for this (following the docs of course), however no acceess or error logs are showing up. Now it probably isn't an issue regarding Graylog, which is why I ask this question here and not on the graylog forum.
Here's the configuration file:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

# Where to look for logs
filebeat.inputs:
- input_type: log
  paths:
    - /var/log/*.log
  type: log

# Where to send the logs 
output.logstash:
   hosts: ["x.x.x.x:5044"]

# Where to store the logs (locally)
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

# Optional modules
filebeat.modules:
- module: apache
  access:
    enabled: true
    var.paths: ["/var/log/httpd/*access.log"]
  error:
    enabled: true
    var.paths: ["/var/log/httpd/*error.log"]

I have checked this configuration file multiple times and there's nothing wrong with it (syntax wise), I have also checked the logs and there are no errors as well. I've made sure that the access and error logs are being filled up with new traffic so that it doesn't try to read empty files. So I really don't know what else it could be.
Aside from the module, everything seems to be logging just fine.

2

Hi @cesq Welcome to the community!

You are sending the data through logstash did you also follow these documents

You did not share your log stash configuration.

Pro tip always send data first directly from filebeat to elasticsearch first and make sure all the data is correct and then put logstash in the middle if you want to.

Follow the filebeat quickstart module first.

Then reconfigure file beat to point to logstash.

Also run full setup not just the dashboards

filebeat setup -e

You do not need logstash, but it's a perfectly valid architecture.

There are lots of topics on this form about the subject.

I think this is one

3

Hi @stephenb, thank you for replying.

I am using Graylog Sidecar feature to run the configs, meaning there's no fixed configuration on the host that the logs are being retrieved from.

Filebeat is installed on the host, but the configs there are irrelevant.
I don't know if you're familiar with how Sidecars work in Graylog, but basically it's a feature that allows me to push one configuration to n hosts. So there's only one config file in play, and that's the one I posted.

I know nothing about this logstash you're referring to, I've gotten filebeat and auditbeat working, only by using the Sidecar feature (and a Beats input). In both examples, nothing is being sent to elasticsearch. Also I want to emphasise that filebeat is working , it's only the apache module that I've tried to get running and failed, no errors in the logs and everything else works fine.

4

This is why I assumed logstash that the logstash output it appears that is what greylog uses as an endpoint

I did a quick search...

greylog filebeat modules

Looks like filebeat modules are not supported... that is unfortunate as they are quite powerful

I suspect the reason they do not work greylog actual workings / requirements of modules. Filebeat modules load / use specific ingest pipelines (parsers) to parse the data that are loaded into elasticsearch and used. Those are loaded into elasticsearch.. I'm pretty certain they are not loaded into greylog.

Why greylog is failing with no errors? I think you're going to need to go to the greylog community and ask them.

I suspect you could just load them as regular logs but I don't know what that looks like on greylog side

5

Ah, that's a shame, thank you for helping me out!

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.