Multiple pipelines with docker not working

Using the following compose service to spin up an ELK stack

  logstash:
    build:
      context: logstash/
    volumes:
      - ./logstash/pipelines.yml:/etc/logstash/pipelines.yml:ro
      # - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:rw
      - ./logstash/pipeline-performance:/usr/share/logstash/pipeline-performance:rw
      - ./logstash/files:/usr/share/logstash/files:rw
      - /datastore/esa_data/synthetic-datasets:/usr/share/logstash/synthetic-datasets
      - /datastore/esa_data/input-datasets:/usr/share/logstash/input-datasets
    ports:
      - "5000:5000"
    environment:
      LS_JAVA_OPTS: "-Xmx4096m -Xms4096m"
    networks:
      - elk
    depends_on:
      - elasticsearch

Here is the contents of pipelines.yml

- pipeline.id: p_performance
  path.config: "/usr/share/logstash/pipeline-performance/"

The directory contains 3 conf files.

The above pipeline never starts;

any idea why?

Does it have to do with the fact I am pointing a pipeline to a folder while perhaps it accepts only .conf files as hinted here?

If the pipeline doesn't start I'd expect the Logstash log to contains clues.

It says "main" pipeline started (I think in case of multiple pipelines it is expected to print the exact pipeline name, in this case (p_performance)

Yes, I believe it's supposed to list all loaded pipelines. Then I'm not sure what's going on.

Your pipelines.yml is mounted into /etc/logstash but the image doesn't look for it there. The /etc/logstash directory is used by the RPM and DEB distributions, but the Docker image is based on the distro-agnostic tarball distribution that keeps all files under the same root. The correct place to mount pipelines.yml in the Docker image is at /usr/share/logstash/config/pipelines.yml. Logstash should find it if you mount it there.

https://www.elastic.co/guide/en/logstash/current/dir-layout.html#docker-layout

2 Likes

Yep, many thanks for this;

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.