# Filebeat modules pipeline selection

**URL:** <https://discuss.elastic.co/t/filebeat-modules-pipeline-selection/160320>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [December 11, 2018, 8:20am UTC](https://discuss.elastic.co/t/filebeat-modules-pipeline-selection/160320 "2018-12-11T08:20:46Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![coudenysj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/coudenysj/32/23108_2.png) [@coudenysj](https://discuss.elastic.co/u/coudenysj)\
**Post date:** [December 11, 2018, 8:20am UTC](https://discuss.elastic.co/t/filebeat-modules-pipeline-selection/160320/1 "2018-12-11T08:20:46Z")

</div>

I was wondering how the filebeat modules create and select the pipeline names.

I have these in my ES setup:

- filebeat-6.4.3-nginx-access-default
- filebeat-6.4.3-mysql-slowlog-pipeline
- etc...

And I can see the content of the pipelines ([https://github.com/elastic/beats/blob/master/filebeat/module/mysql/slowlog/ingest/pipeline.json](https://github.com/elastic/beats/blob/master/filebeat/module/mysql/slowlog/ingest/pipeline.json)), but I can't find where the given pipeline is selected when shipping data into Elasticsearch.

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [December 11, 2018, 7:08pm UTC](https://discuss.elastic.co/t/filebeat-modules-pipeline-selection/160320/2 "2018-12-11T19:08:44Z")

</div>

Filebeat has a concept of modules (`nginx`, `mysql`, etc.). Each module has one or more filesets (e.g. `nginx/access`, `nginx/error`, `mysql/slowlog`, etc.).

In each fileset's source code there is a `manifest.yml` file. This file defines (amongst other things) the pipeline to be used for the fileset. Here are some examples:

- [https://github.com/elastic/beats/blob/master/filebeat/module/nginx/access/manifest.yml#L12](https://github.com/elastic/beats/blob/master/filebeat/module/nginx/access/manifest.yml#L12)
- [https://github.com/elastic/beats/blob/master/filebeat/module/nginx/error/manifest.yml#L12](https://github.com/elastic/beats/blob/master/filebeat/module/nginx/error/manifest.yml#L12)
- [https://github.com/elastic/beats/blob/master/filebeat/module/mysql/slowlog/manifest.yml#L13](https://github.com/elastic/beats/blob/master/filebeat/module/mysql/slowlog/manifest.yml#L13)

Note that sometimes the pipeline might be chosen more dynamically based on some configuration value in the fileset's configuration. For example:

- [https://github.com/elastic/beats/blob/master/filebeat/module/logstash/slowlog/manifest.yml#L12](https://github.com/elastic/beats/blob/master/filebeat/module/logstash/slowlog/manifest.yml#L12)

Hope that helps,

Shaunak

---

<div class="post-metadata">

**Author:** ![coudenysj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/coudenysj/32/23108_2.png) [@coudenysj](https://discuss.elastic.co/u/coudenysj)\
**Post date:** [December 12, 2018, 9:07am UTC](https://discuss.elastic.co/t/filebeat-modules-pipeline-selection/160320/3 "2018-12-12T09:07:59Z")

</div>

Hi @shaunak, thanks for the reply!

I see the file locations with the content of the pipelines, but I can't figure out where the pipeline names come from (filebeat-6.4.3-nginx-access-default, filebeat-6.4.3-mysql-slowlog-pipeline, etc...) and more importantly, how they are selected when Filebeat is shipping logs to Elasticsearch :).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 9, 2019, 9:08am UTC](https://discuss.elastic.co/t/filebeat-modules-pipeline-selection/160320/4 "2019-01-09T09:08:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.