# Multiple pipelines with Fileabeat and Logstash

**URL:** https://discuss.elastic.co/t/multiple-pipelines-with-fileabeat-and-logstash/301652
**Category:** Logstash
**Created:** [April 5, 2022, 3:17pm UTC](https://discuss.elastic.co/t/multiple-pipelines-with-fileabeat-and-logstash/301652 "2022-04-05T15:17:29Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![bianca6](https://avatars.discourse-cdn.com/v4/letter/b/85e7bf/32.png) [@bianca6](https://discuss.elastic.co/u/bianca6)
#### Post date: [April 5, 2022, 3:17pm UTC](https://discuss.elastic.co/t/multiple-pipelines-with-fileabeat-and-logstash/301652/1 "2022-04-05T15:17:29Z")

</div>

Hi,

I know this is a known topic, I read different discussions but.. I'm still stuck, if someone could help me I will be grateful.

What I'm trying to achieve is this:  
[https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html#distributor-pattern](https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html#distributor-pattern)

My Filebeat configuration is listening to different folders, and it does react as soon as I add a file in one of them:

```auto
filebeat.inputs:

- type: log
  enabled: true
  paths:
    - d:/mygit/all/*.log
  
- type: log
  enabled: true
  paths: 
    - d:/mygit/first/*.log
  tags: ["first"]

- type: log
  enabled: true
  paths:
    - d:/mygit/second/*.log
  tags: ["second"]

```

And here is my pipelines.yml:

```auto
- pipeline.id: beats-server
  config.string: |
    input { beats { port => 5044 } }
    output {
        if "first" in [tags] {
          pipeline { send_to => first}
        } else if "second" in [tags] {
          pipeline { send_to => second}
        } else {
          pipeline { send_to => all}
        }
    }

- pipeline.id: first
  path.config: "d:/mygit/my-config-files/first.conf"

- pipeline.id: second
  path.config: "d:/mygit/my-config-files/second.conf"

- pipeline.id: all
  path.config: "d:/mygit/all.conf"

```

And my configuration files start with (same for first, second, all):

```auto
input { pipeline { address => first} }

```

As soon as I drag and drop a file in the filebeat's folders, I have an error on my Logstash terminal:

```auto
[2022-04-05T17:02:06,459][WARN][org.logstash.plugins.pipeline.PipelineBus][beats-server][<long id>] Attempted to send event to 'first' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.

```

To start Logstash I just did:

```auto
./bin/logstash

```

The pipeline.yml is automaticaly chose, and everything is running as usual, the last lines (before I add a file) are:

```auto
[2022-04-05T17:01:45,500][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:"beats-server"], :non_running_pipelines=>[]}
[2022-04-05T17:01:45,546][INFO][org.logstash.beats.Server][beats-server][<long id>] Starting server on port: 5044

```

What am I doing wrong? Do I have to run multiple Logstash for the different pipelines? Or something else like this?

Thanks!

---

<div class="post-metadata">

### Author: ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [April 5, 2022, 3:38pm UTC](https://discuss.elastic.co/t/multiple-pipelines-with-fileabeat-and-logstash/301652/2 "2022-04-05T15:38:43Z")

</div>

> [@bianca6](#):
>
> `running_pipelines=>[:"beats-server"]`

Your logstash is starting only the first pipeline, called `beats-server` in the `pipelines.yml` file, it probably couldn't find the other pipelines.

I think this is a Windows issue with the path.

Can you try to change the `path.config` for something like this for your other pipelines?

```auto
path.config: "/D:/mygit/my-config-files/first.conf"

```

This is a working around that is said to work according to this [comment](https://github.com/elastic/logstash/issues/9144#issuecomment-423061019) on a github issue.

---

<div class="post-metadata">

### Author: ![bianca6](https://avatars.discourse-cdn.com/v4/letter/b/85e7bf/32.png) [@bianca6](https://discuss.elastic.co/u/bianca6)
#### Post date: [April 6, 2022, 7:38am UTC](https://discuss.elastic.co/t/multiple-pipelines-with-fileabeat-and-logstash/301652/3 "2022-04-06T07:38:38Z")

</div>

Hi! Thanks a lot, it is working now!

```auto
[2022-04-06T09:30:03,860][INFO][logstash.agent] Pipelines running {:count=>4, :running_pipelines=>[:"beats-server", :first, :all, :second], :non_running_pipelines=>[]}

```

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 4, 2022, 7:39am UTC](https://discuss.elastic.co/t/multiple-pipelines-with-fileabeat-and-logstash/301652/4 "2022-05-04T07:39:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.