# Multiple filebeat instances in single docker container?

**URL:** <https://discuss.elastic.co/t/multiple-filebeat-instances-in-single-docker-container/177335>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [April 17, 2019, 3:51pm UTC](https://discuss.elastic.co/t/multiple-filebeat-instances-in-single-docker-container/177335 "2019-04-17T15:51:27Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![JJB](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jjb/32/44079_2.png) [@JJB](https://discuss.elastic.co/u/JJB)\
**Post date:** [April 17, 2019, 3:51pm UTC](https://discuss.elastic.co/t/multiple-filebeat-instances-in-single-docker-container/177335/1 "2019-04-17T15:51:28Z")

</div>

I am running 4 docker containers for ELKB. I have a Filebeat set up to bring in log files from a volume folder and a logstash filter to process the log files (they're of the same format).

I now have another set of log files in a different format. I've created a second pipeline with it's own .conf file and filter.

But this means I have two pipelines outputting to elasticsearch from the same input.. port 5044 - problem 1.

My filebeat.yml for the original set of log files looks like this:

```
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/share/logs/**/*.log # docker volume with many logs in folders
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
 ...
output.logstash:
  hosts: ["logstash:5044"]

```

But the other log files have a different format so the multiline pattern doesn't work - problem 2.

Ok, so I could create a second `-type: log` section, but I still have the problem that the filebeat.yml contains a single `output.logstash` to 5044 - back to problem 1.

So my thought was that I need one filebeat per pipeline... but if so, how do I run multiple filebeats on a single docker container. And is this the right thing to do?

Could I use tags and a single pipeline (with if/else in pipeline logstash.conf file)?

Apologies if this question isn't quite clear. I'm fairly new to Docker and Elastic so have had a steep learning curve in last couple of weeks.

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [April 17, 2019, 5:32pm UTC](https://discuss.elastic.co/t/multiple-filebeat-instances-in-single-docker-container/177335/2 "2019-04-17T17:32:50Z")

</div>

I think you are on the right track with creating a second `log` input, using tags, and if/else in your Logstash pipeline.

Under each `log` input in your `filebeat.yml` you can specify an [`add_tags` processor](https://www.elastic.co/guide/en/beats/filebeat/current/add-tags.html) to set a unique tag for that input. Then, in the Logstash pipeline, you can use if/else on the tags to process the events differently.

---

<div class="post-metadata">

**Author:** ![JJB](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jjb/32/44079_2.png) [@JJB](https://discuss.elastic.co/u/JJB)\
**Post date:** [April 17, 2019, 7:32pm UTC](https://discuss.elastic.co/t/multiple-filebeat-instances-in-single-docker-container/177335/3 "2019-04-17T19:32:12Z")

</div>

Thanks Shaunak, so it seems I would have one logstash.conf with if/else which I guess is ok, it just seems to go against everything I've read in logstash so far which urges you to create a pipeline per the [Multiple Pipelines documentation](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html)

> "Using multiple pipelines is especially useful if your current configuration has event flows that don’t share the same inputs/filters and outputs and are being separated from each other using tags and conditionals"

I guess my logs do share the same output, and only really share the same input because it's in the one filebeat.yml, but the actual `filebeats.inputs` are different and the filters are certainly different.

I think it would be really useful to document this approach in the Elastic docs, as I've seen a few other people asking similar questions but not quite conclusive solutions, and I think it's a common need to have a central ELKB service harvesting logs from many services with different formats (to my novice eyes anyway!)

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [April 17, 2019, 9:10pm UTC](https://discuss.elastic.co/t/multiple-filebeat-instances-in-single-docker-container/177335/4 "2019-04-17T21:10:05Z")

</div>

I think it really depends on the complexity of your Logstash pipeline. If there are a lot of plugins that are common and just a few differences that require the if/else, then perhaps one pipeline is better. But if the differences are greater, then perhaps using multiple pipelines with [pipeline-to-pipeline communication](https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html) might be preferred. So I think it comes down to what you, the pipeline author and maintainer, consider to be clean vs. messy from a readability and maintenance point-of-view.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 15, 2019, 9:10pm UTC](https://discuss.elastic.co/t/multiple-filebeat-instances-in-single-docker-container/177335/5 "2019-05-15T21:10:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.