So I have been playing around with collecting data using filebeat and sending it via sidecar to my graylog server. I wanted to try out the apache module, so I wrote the configuration for this (following the docs of course), however no acceess or error logs are showing up. Now it probably isn't an issue regarding Graylog, which is why I ask this question here and not on the graylog forum.
Here's the configuration file:
# Needed for Graylog
# Where to look for logs
- input_type: log
# Where to send the logs
# Where to store the logs (locally)
# Optional modules
- module: apache
I have checked this configuration file multiple times and there's nothing wrong with it (syntax wise), I have also checked the logs and there are no errors as well. I've made sure that the access and error logs are being filled up with new traffic so that it doesn't try to read empty files. So I really don't know what else it could be.
Aside from the module, everything seems to be logging just fine.
I am using Graylog Sidecar feature to run the configs, meaning there's no fixed configuration on the host that the logs are being retrieved from.
Filebeat is installed on the host, but the configs there are irrelevant.
I don't know if you're familiar with how Sidecars work in Graylog, but basically it's a feature that allows me to push one configuration to n hosts. So there's only one config file in play, and that's the one I posted.
I know nothing about this logstash you're referring to, I've gotten filebeat and auditbeat working, only by using the Sidecar feature (and a Beats input). In both examples, nothing is being sent to elasticsearch. Also I want to emphasise that filebeat is working , it's only the apache module that I've tried to get running and failed, no errors in the logs and everything else works fine.
This is why I assumed logstash that the logstash output it appears that is what greylog uses as an endpoint
I did a quick search...
greylog filebeat modules
Looks like filebeat modules are not supported... that is unfortunate as they are quite powerful
I suspect the reason they do not work greylog actual workings / requirements of modules. Filebeat modules load / use specific ingest pipelines (parsers) to parse the data that are loaded into elasticsearch and used. Those are loaded into elasticsearch.. I'm pretty certain they are not loaded into greylog.
Why greylog is failing with no errors? I think you're going to need to go to the greylog community and ask them.
I suspect you could just load them as regular logs but I don't know what that looks like on greylog side