# Multiple pipelines with docker not working

**URL:** <https://discuss.elastic.co/t/multiple-pipelines-with-docker-not-working/145181>\
**Category:** Logstash\
**Created:** [August 20, 2018, 2:03pm UTC](https://discuss.elastic.co/t/multiple-pipelines-with-docker-not-working/145181 "2018-08-20T14:03:36Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![pkaramol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pkaramol/32/22610_2.png) [@pkaramol](https://discuss.elastic.co/u/pkaramol)\
**Post date:** [August 20, 2018, 2:03pm UTC](https://discuss.elastic.co/t/multiple-pipelines-with-docker-not-working/145181/1 "2018-08-20T14:03:36Z")

</div>

Using the following compose service to spin up an `ELK` stack

```
  logstash:
    build:
      context: logstash/
    volumes:
      - ./logstash/pipelines.yml:/etc/logstash/pipelines.yml:ro
      # - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:rw
      - ./logstash/pipeline-performance:/usr/share/logstash/pipeline-performance:rw
      - ./logstash/files:/usr/share/logstash/files:rw
      - /datastore/esa_data/synthetic-datasets:/usr/share/logstash/synthetic-datasets
      - /datastore/esa_data/input-datasets:/usr/share/logstash/input-datasets
    ports:
      - "5000:5000"
    environment:
      LS_JAVA_OPTS: "-Xmx4096m -Xms4096m"
    networks:
      - elk
    depends_on:
      - elasticsearch

```

Here is the contents of `pipelines.yml`

```
- pipeline.id: p_performance
  path.config: "/usr/share/logstash/pipeline-performance/"

```

The directory contains 3 conf files.

The above pipeline never starts;

any idea why?

Does it have to do with the fact I am pointing a pipeline to a folder while perhaps it accepts only `.conf` files as hinted [here](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html)?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 20, 2018, 2:08pm UTC](https://discuss.elastic.co/t/multiple-pipelines-with-docker-not-working/145181/2 "2018-08-20T14:08:15Z")

</div>

If the pipeline doesn't start I'd expect the Logstash log to contains clues.

---

<div class="post-metadata">

**Author:** ![pkaramol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pkaramol/32/22610_2.png) [@pkaramol](https://discuss.elastic.co/u/pkaramol)\
**Post date:** [August 20, 2018, 2:10pm UTC](https://discuss.elastic.co/t/multiple-pipelines-with-docker-not-working/145181/3 "2018-08-20T14:10:28Z")

</div>

It says "main" pipeline started (I think in case of multiple pipelines it is expected to print the exact pipeline name, in this case (`p_performance`)

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 20, 2018, 2:15pm UTC](https://discuss.elastic.co/t/multiple-pipelines-with-docker-not-working/145181/4 "2018-08-20T14:15:05Z")

</div>

Yes, I believe it's supposed to list all loaded pipelines. Then I'm not sure what's going on.

---

<div class="post-metadata">

**Author:** ![jarpy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jarpy/32/51290_2.png) [@jarpy](https://discuss.elastic.co/u/jarpy)\
**Post date:** [August 21, 2018, 1:18am UTC](https://discuss.elastic.co/t/multiple-pipelines-with-docker-not-working/145181/5 "2018-08-21T01:18:57Z")

</div>

Your `pipelines.yml` is mounted into `/etc/logstash` but the image doesn't look for it there. The `/etc/logstash` directory is used by the RPM and DEB distributions, but the Docker image is based on the distro-agnostic tarball distribution that keeps all files under the same root. The correct place to mount `pipelines.yml` in the Docker image is at `/usr/share/logstash/config/pipelines.yml`. Logstash should find it if you mount it there.

[https://www.elastic.co/guide/en/logstash/current/dir-layout.html#docker-layout](https://www.elastic.co/guide/en/logstash/current/dir-layout.html#docker-layout)

---

<div class="post-metadata">

**Author:** ![pkaramol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pkaramol/32/22610_2.png) [@pkaramol](https://discuss.elastic.co/u/pkaramol)\
**Post date:** [August 22, 2018, 7:46am UTC](https://discuss.elastic.co/t/multiple-pipelines-with-docker-not-working/145181/6 "2018-08-22T07:46:22Z")

</div>

Yep, many thanks for this;

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 19, 2018, 7:46am UTC](https://discuss.elastic.co/t/multiple-pipelines-with-docker-not-working/145181/7 "2018-09-19T07:46:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.