Filebeat modules pipeline selection

[coudenysj (Jachim Coudenys)]

I was wondering how the filebeat modules create and select the pipeline names.

I have these in my ES setup:

• filebeat-6.4.3-nginx-access-default
• filebeat-6.4.3-mysql-slowlog-pipeline
• etc...

And I can see the content of the pipelines (https://github.com/elastic/beats/blob/master/filebeat/module/mysql/slowlog/ingest/pipeline.json), but I can't find where the given pipeline is selected when shipping data into Elasticsearch.

[shaunak (Shaunak Kashyap)]

Filebeat has a concept of modules (nginx, mysql, etc.). Each module has one or more filesets (e.g. nginx/access, nginx/error, mysql/slowlog, etc.).

In each fileset's source code there is a manifest.yml file. This file defines (amongst other things) the pipeline to be used for the fileset. Here are some examples:

• https://github.com/elastic/beats/blob/master/filebeat/module/nginx/access/manifest.yml#L12
• https://github.com/elastic/beats/blob/master/filebeat/module/nginx/error/manifest.yml#L12
• https://github.com/elastic/beats/blob/master/filebeat/module/mysql/slowlog/manifest.yml#L13

Note that sometimes the pipeline might be chosen more dynamically based on some configuration value in the fileset's configuration. For example:

• https://github.com/elastic/beats/blob/master/filebeat/module/logstash/slowlog/manifest.yml#L12

Hope that helps,

Shaunak

[coudenysj (Jachim Coudenys)]

Hi @shaunak, thanks for the reply!

I see the file locations with the content of the pipelines, but I can't figure out where the pipeline names come from (filebeat-6.4.3-nginx-access-default, filebeat-6.4.3-mysql-slowlog-pipeline, etc...) and more importantly, how they are selected when Filebeat is shipping logs to Elasticsearch :).