Different modules?

Elastic Stack Beats
,
1

We're using filebeat to ingest all kinds of logfiles: application logs, apache, nginx, whatever.

For specifix log files, like apache, we want to parse the message into different fields. The apache module with its ingest node pipeline seems to be the correct way.

But we don't want to apply this pipeline to ALL messages, only to apache logs. Is there a way to define how to apply which module to which log file?

2

According to the documentation, you have to specify the paths monitored by the module (in module dedicated config file), so it should work in parallel with your classic input defined in the dedicated file.

3

As far as I understand, modules are a combination of
a) default input configuration (covered by the paths monitored)
b) ingest node pipelines
c) Kibana dashboards.

Ingest node pipelines are in elasticsearch. I can list them at http://:9200/_ingest/pipeline
But they seem to be globally? Or does Filebeat select pipelines conditionally during ingest? How can I confirm that?

4

Well try to run both module and config file and count the number of filebeat process, this should indicate if they are separated or not.
The module will use his default ingest pipeline but in normal configuration you can select another target in output, aka index pattern or pipeline id.
Didn't try that in filebeat but worked few years ago with logstash standard pipeline + netflow module.
Please tell me if you find out :slight_smile:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.