# Logstash 6.2 Multiple Pipelines,each with its own output file

**URL:** <https://discuss.elastic.co/t/logstash-6-2-multiple-pipelines-each-with-its-own-output-file/127547>\
**Category:** Logstash\
**Created:** [April 10, 2018, 8:44pm UTC](https://discuss.elastic.co/t/logstash-6-2-multiple-pipelines-each-with-its-own-output-file/127547 "2018-04-10T20:44:27Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![davehatma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davehatma/32/29322_2.png) [@davehatma](https://discuss.elastic.co/u/davehatma)\
**Post date:** [April 10, 2018, 8:44pm UTC](https://discuss.elastic.co/t/logstash-6-2-multiple-pipelines-each-with-its-own-output-file/127547/1 "2018-04-10T20:44:28Z")

</div>

I'm running three pipelines from 'pipelines.yml' each pipeline config file has its own output filter:

> ```
> file {
> path => "/var/log/logstash/logstash-[my-pipeline name].log"
> }
> 
> ```

I'm finding that the output for all three configs is going **to just one** of the log files and appears to be the first one in pipelines.yml.

Is this expected behavior? If not is there a bug? If so, I think it would be a nice feature.

I like to keep the log files for each pipeline separate as they are easier to search and manage.

Thanks in advance,

Dave Howell

---

<div class="post-metadata">

**Author:** ![Evesy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evesy/32/29520_2.png) [@Evesy](https://discuss.elastic.co/u/Evesy)\
**Post date:** [April 10, 2018, 10:24pm UTC](https://discuss.elastic.co/t/logstash-6-2-multiple-pipelines-each-with-its-own-output-file/127547/2 "2018-04-10T22:24:59Z")

</div>

Hi Dave,

Would you be able to share the contents of `pipelines.yml` & the respective configs that it references?

Providing your Logstash output is pointing to a different path in each pipeline I see no reason all your outputs would end up in the same file

---

<div class="post-metadata">

**Author:** ![Suman\_Reddy1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suman_reddy1/32/27303_2.png) [@Suman\_Reddy1](https://discuss.elastic.co/u/Suman_Reddy1)\
**Post date:** [April 11, 2018, 3:37am UTC](https://discuss.elastic.co/t/logstash-6-2-multiple-pipelines-each-with-its-own-output-file/127547/3 "2018-04-11T03:37:28Z")

</div>

Also, please paste your logstash startup commands

---

<div class="post-metadata">

**Author:** ![davehatma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davehatma/32/29322_2.png) [@davehatma](https://discuss.elastic.co/u/davehatma)\
**Post date:** [April 11, 2018, 2:24pm UTC](https://discuss.elastic.co/t/logstash-6-2-multiple-pipelines-each-with-its-own-output-file/127547/4 "2018-04-11T14:24:11Z")

</div>

Most certainly.

I'm using systemd to start /stop Logstash. Note that when I use the command line  
instead (/usr/share/logstash/bin/logstash), I get the same behavior.

Here is the logstash.system file:

```
[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target

```

Here is the content of pipelines.yml with all commented lines removed:

```
- pipeline.id: metrics
  path.config: "/etc/logstash/conf.d/metrics.conf"
- pipeline.id: exceptions
  path.config: "/etc/logstash/conf.d/exceptions.conf"
- pipeline.id: transactions
  path.config: "/etc/logstash/conf.d/transactions.conf"

```

As I am unable to paste in the full content of the three config files (due to character limit for responding in the forum), here are the "output" filters for the three config files (which should be the only pertinent sections for the behavior I am experiencing):

metrics.conf:

```
output {
    elasticsearch {
        action => "index"
        hosts => "elasticlog-p01:9200"
        index => "%{app_group}::%{app_name}::metrics::%{[@metadata][index_env]}%{[@metadata][index_log_date]}"
    }

    file {
        path => "/var/log/logstash/logstash-metrics.log"
    }
}

```

exceptions.conf:

```
output {
    elasticsearch {
        action => "index"
        hosts => "elasticlog-p01:9200"
        index => "%{app_group}::%{app_name}::metrics::%{[@metadata][index_env]}%{[@metadata][index_log_date]}"
    }

    file {
        path => "/var/log/logstash/logstash-exceptions.log"
    }
}

```

transactions.conf:

```
output {
    elasticsearch {
        action => "index"
        hosts => "elasticlog-p01:9200"
        index => "%{app_group}::%{app_name}::transactions::%{[@metadata][index_env]}%{[@metadata][index_log_date]}"
    }
    
    file {
        path => "/var/log/logstash/logstash-transactions.log"
    }
}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 11, 2018, 2:30pm UTC](https://discuss.elastic.co/t/logstash-6-2-multiple-pipelines-each-with-its-own-output-file/127547/5 "2018-04-11T14:30:40Z")

</div>

Is your exceptions output really the same as the metrics output, or did you paste in the wrong config?

---

<div class="post-metadata">

**Author:** ![davehatma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davehatma/32/29322_2.png) [@davehatma](https://discuss.elastic.co/u/davehatma)\
**Post date:** [April 11, 2018, 2:48pm UTC](https://discuss.elastic.co/t/logstash-6-2-multiple-pipelines-each-with-its-own-output-file/127547/6 "2018-04-11T14:48:21Z")

</div>

No they are different. Paste error on my part. It should be '.../exceptions.log'. I have edited my reply to reflect. Thanks for pointing that out.

---

<div class="post-metadata">

**Author:** ![davehatma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davehatma/32/29322_2.png) [@davehatma](https://discuss.elastic.co/u/davehatma)\
**Post date:** [April 11, 2018, 3:23pm UTC](https://discuss.elastic.co/t/logstash-6-2-multiple-pipelines-each-with-its-own-output-file/127547/7 "2018-04-11T15:23:08Z")

</div>

Michael and Suman, with apologies please disregard this issue. I don't know how or why, but I came in this morning and it is working now. All three logs are being posted as desired.

Thanks for you help and sorry for the false alarm.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 9, 2018, 3:23pm UTC](https://discuss.elastic.co/t/logstash-6-2-multiple-pipelines-each-with-its-own-output-file/127547/8 "2018-05-09T15:23:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.