# \[Help\] Multiple pipelines with N logs from filebeat and N servers

**URL:** <https://discuss.elastic.co/t/help-multiple-pipelines-with-n-logs-from-filebeat-and-n-servers/179159>\
**Category:** Logstash\
**Created:** [April 30, 2019, 10:47pm UTC](https://discuss.elastic.co/t/help-multiple-pipelines-with-n-logs-from-filebeat-and-n-servers/179159 "2019-04-30T22:47:02Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![TiNhO](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tinho/32/41986_2.png) [@TiNhO](https://discuss.elastic.co/u/TiNhO)\
**Post date:** [April 30, 2019, 10:47pm UTC](https://discuss.elastic.co/t/help-multiple-pipelines-with-n-logs-from-filebeat-and-n-servers/179159/1 "2019-04-30T22:47:02Z")

</div>

I have this structure:

```
                                  ____________________  ______________
                                 |same name | |same name | | same name |
                                 |----------| |----------| |--------------|
Machine1 > fileabeat with 3 log (| logA.txt |, | logB.log | and |logC.something|)
Machine2 > fileabeat with 3 log (| logA.txt |, | logB.log | and |logC.something|)
Machine3 > fileabeat with 3 log (| logA.txt |, | logB.log | and |logC.something|)
... | | | | | |
Machine6 > fileabeat with 3 log (| logA.txt |, | logB.log | and |logC.something|)
                                 |----------| |----------| |--------------|

```

The output filebeat is logstash on port 5044 for all filebeat.

My logstash are on other machine and this machine have too elasticsearch and kibana.  
On logstash I made three _ **.conf** _ using beat input on 5044 port with grok filter made by my self and as output is elasticsearch.  
My **logstash.yml** have this config:

```
  pipeline.batch.size: 125
  pipeline.batch.delay: 5
- pipeline.id: dbjus
  pipeline.workers: 1
  path.config: "/path/to/logA.conf"
- pipeline.id: tasks
  pipeline.workers: 1
  path.config: "/path/to/logB.log"
- pipeline.id: info_log
  pipeline.workers: 1
  path.config: "/path/to/logC.something"
  http.host: "127.0.0.1"
  http.port: 9600

```

And my **pipeline.yml** have this:

```
 - pipeline.id: logA
   path.config: "/path/to/logA.conf"
 - pipeline.id: logB
   path.config: "/path/to/logB.conf"
 - pipeline.id: logC
   path.config: "/path/to/logC.conf"

```

Before this configuration I tested each .conf with only one machine and worked fine. Kibana show me the logs. But when use the configuration above does not work! Kibana do not show the logs.

Probably I do something wrong.

PS.: Sorry for my English. I'm not a speaker of English.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 1, 2019, 5:51am UTC](https://discuss.elastic.co/t/help-multiple-pipelines-with-n-logs-from-filebeat-and-n-servers/179159/2 "2019-05-01T05:51:21Z")

</div>

Multiple pipelines can not have inputs using the same port so I would recommend creating a single input pipeline where you have a beats input listening to port 5044. In this pipeline you can then look at the data and determine which pipeline should process it before sending it to the correct pipeline using [pipeline-to-pipeline communication](https://www.elastic.co/guide/en/logstash/7.0/pipeline-to-pipeline.html).

---

<div class="post-metadata">

**Author:** ![TiNhO](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tinho/32/41986_2.png) [@TiNhO](https://discuss.elastic.co/u/TiNhO)\
**Post date:** [May 2, 2019, 2:41pm UTC](https://discuss.elastic.co/t/help-multiple-pipelines-with-n-logs-from-filebeat-and-n-servers/179159/3 "2019-05-02T14:41:49Z")

</div>

[I hope this is my final edit! =D]

I think understood. I will use the distributor patter with some ifs with conf.string. But the example in documentation use [type], I need use the path log. What I need put instead of [type] to use the path log?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 30, 2019, 2:48pm UTC](https://discuss.elastic.co/t/help-multiple-pipelines-with-n-logs-from-filebeat-and-n-servers/179159/4 "2019-05-30T14:48:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.