Multiple pipelines with docker not working

[pkaramol (Pantelis Karamolegkos)]

Using the following compose service to spin up an ELK stack

  logstash:
    build:
      context: logstash/
    volumes:
      - ./logstash/pipelines.yml:/etc/logstash/pipelines.yml:ro
      # - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:rw
      - ./logstash/pipeline-performance:/usr/share/logstash/pipeline-performance:rw
      - ./logstash/files:/usr/share/logstash/files:rw
      - /datastore/esa_data/synthetic-datasets:/usr/share/logstash/synthetic-datasets
      - /datastore/esa_data/input-datasets:/usr/share/logstash/input-datasets
    ports:
      - "5000:5000"
    environment:
      LS_JAVA_OPTS: "-Xmx4096m -Xms4096m"
    networks:
      - elk
    depends_on:
      - elasticsearch

Here is the contents of pipelines.yml

- pipeline.id: p_performance
  path.config: "/usr/share/logstash/pipeline-performance/"

The directory contains 3 conf files.

The above pipeline never starts;

any idea why?

Does it have to do with the fact I am pointing a pipeline to a folder while perhaps it accepts only .conf files as hinted here?

[magnusbaeck (Magnus Bäck)]

If the pipeline doesn't start I'd expect the Logstash log to contains clues.

[pkaramol (Pantelis Karamolegkos)]

It says "main" pipeline started (I think in case of multiple pipelines it is expected to print the exact pipeline name, in this case (p_performance)

[magnusbaeck (Magnus Bäck)]

Yes, I believe it's supposed to list all loaded pipelines. Then I'm not sure what's going on.

[jarpy (Nina Dowland)]

Your pipelines.yml is mounted into /etc/logstash but the image doesn't look for it there. The /etc/logstash directory is used by the RPM and DEB distributions, but the Docker image is based on the distro-agnostic tarball distribution that keeps all files under the same root. The correct place to mount pipelines.yml in the Docker image is at /usr/share/logstash/config/pipelines.yml. Logstash should find it if you mount it there.

https://www.elastic.co/guide/en/logstash/current/dir-layout.html#docker-layout

[pkaramol (Pantelis Karamolegkos)]

Yep, many thanks for this;