# Coupling Filebeat prospector with Logstash pipeline

**URL:** <https://discuss.elastic.co/t/coupling-filebeat-prospector-with-logstash-pipeline/108349>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 20, 2017, 9:19am UTC](https://discuss.elastic.co/t/coupling-filebeat-prospector-with-logstash-pipeline/108349 "2017-11-20T09:19:14Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![dba](https://avatars.discourse-cdn.com/v4/letter/d/cdc98d/32.png) [@dba](https://discuss.elastic.co/u/dba)\
**Post date:** [November 20, 2017, 9:19am UTC](https://discuss.elastic.co/t/coupling-filebeat-prospector-with-logstash-pipeline/108349/1 "2017-11-20T09:19:14Z")

</div>

Hi

We are moving to ELK 6.0 and are very pleased that we can now define filebeat prospectors and logstash pipelines in their own files. This makes it easier for our teams to create and deploy their own configurations (and have filebeat and logstash dynamically reload the configuration).

We have filebeat running with multiple prospectors, but a prospector can't define it's own output. So it looks like we are forced to use the same output.  
We have multiple pipelines in logstash, but since all events from filebeat are delivered to the same input, we can't use them.

Googling the problem it seems the recommended solution is to have multiple instances of filebeat, but this is not optimal.

Is there anyway for us to have a prospector deliver to a specific pipeline, without having to alter the "shared" configuration files? Or for logstash to route to a specific pipeline, again without altering the "shared" configuration? Maybe something like adding a tag in the prospector and filtering on that in the "input" of the pipeline?

Best regards  
dba

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [November 21, 2017, 12:33am UTC](https://discuss.elastic.co/t/coupling-filebeat-prospector-with-logstash-pipeline/108349/2 "2017-11-21T00:33:01Z")

</div>

This would require some kind of filtering + forward to another pipeline in Logstash. I don't think this is possible as is with Logstash. e.g. see: [Logstash 6 - Multiple Pipelines for One Input](https://discuss.elastic.co/t/logstash-6-multiple-pipelines-for-one-input/107929)

In filebeat in the prospector you can either use the `fields` setting to pass additional fields for filtering or use the `pipeline` setting in filebeat. The later is somewhat private and used by filebeat to set the ingest node pipeline name for the Elasticsearch output. The `pipeline` setting of each prospector is available in the event via `[@metadata][pipeline]`. You can still have separate processing 'pipelines/filters' so to say, by guarding those on the `pipeline` contents. Each prospector specific processing would be wrapped into this:

```auto
if [@metadata][pipeline] {
  mutate {
    remove_field => ["[@metadata][pipeline]"]
  }

  # custom per prospector filters
}

```

If you really filter on `[@metadata][pipeline]`, make sure to remove it as well. With 6.0 we ask users to set the ingest node pipeline via `[@metadata][pipeline]`. Or rather use a custom field.

---

<div class="post-metadata">

**Author:** ![dba](https://avatars.discourse-cdn.com/v4/letter/d/cdc98d/32.png) [@dba](https://discuss.elastic.co/u/dba)\
**Post date:** [November 21, 2017, 8:30am UTC](https://discuss.elastic.co/t/coupling-filebeat-prospector-with-logstash-pipeline/108349/3 "2017-11-21T08:30:14Z")

</div>

Hi Steffen

Thank you for the reply 🙂

We have a team managing our ELK cluster and we also have a number of software development teams. I would love if a software development team could deploy a service/site and include a filebeat prospector and logstash pipeline (with automatic reloading of configuration and no editing of the shared configuration). This is already possible, but we just need the ability to link the two.  
Is that a feature Elastic would consider adding? (as I understand your suggestion, we can't get all the way where we want to be).

Best regards  
dba

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [November 21, 2017, 12:38pm UTC](https://discuss.elastic.co/t/coupling-filebeat-prospector-with-logstash-pipeline/108349/4 "2017-11-21T12:38:26Z")

</div>

Maybe this is better to be asked in the Logstash forum.

Logstash supports the a `conf.d` folder to load multiple configuration files. See `path.config` setting in logstash.yml file. (e.g.: [Logstash Directory Layout](https://www.elastic.co/guide/en/logstash/current/dir-layout.html)).

Yet I'm not sure if automatic config reloading works with `conf.d`, e.g. by just adding/removing files with filter definitions in conf.d. If so, and given they are all added to the same pipeline in logstash, the if-guard trick shown might work.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 19, 2017, 12:38pm UTC](https://discuss.elastic.co/t/coupling-filebeat-prospector-with-logstash-pipeline/108349/5 "2017-12-19T12:38:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.