Logstash docker pipeline volume not mounting

Hi,
I am hoping someone can help me spot where am I doing a mistake.
I am trying to spin up a simple logstash docker container (oss) with a persistent volume mounted to the default pipeline directory.
However, when I look into the container, I see it only has the default *.conf file in the pipeline directory.

This is my docker-compose:

version: '3'
services:

  logstash:
      image: docker.elastic.co/logstash/logstash-oss:6.6.2
      container_name: logstash
      volumes:
         - lspipeline:/usr/share/logstash/pipeline
     ports:
         - 9600:9600
         - 5040:5040
         - 8181:8181
     networks:
         - esnet

  volumes:
      lspipeline:
  networks:
      esnet:

I can see that logstash starts up using only the default pipeline:

> Creating logstash ... done
> Attaching to logstash
> logstash    | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
> logstash    | [2019-03-24T09:12:42,694][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
> logstash    | [2019-03-24T09:12:42,701][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
> logstash    | [2019-03-24T09:12:43,140][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.6.2"}
> logstash    | [2019-03-24T09:12:43,165][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"63f045c6-f091-4ecb-8685-9a5789ec62db", :path=>"/usr/share/logstash/data/uuid"}
> logstash    | [2019-03-24T09:12:47,939][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
> logstash    | [2019-03-24T09:12:48,194][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
> logstash    | [2019-03-24T09:12:48,217][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x55c7ee72 run>"}
> logstash    | [2019-03-24T09:12:48,262][INFO ][org.logstash.beats.Server] Starting server on port: 5044
> logstash    | [2019-03-24T09:12:48,263][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
> logstash    | [2019-03-24T09:12:48,487][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Login into the container and listing the contents of the pipeline directory (/usr/share/logstash/pipeline):

> bash-4.2$ ls pipeline/
> logstash.conf
> bash-4.2$ 

Inspecting the container, I see the mount is there.
But my mounted pipeline directory has three pipeline configurations. (no logstash.conf)

>         "Mounts": [
>             {
>                 "Type": "volume",
>                 "Name": "mn_lspipeline",
>                 "Source": "/var/lib/docker/volumes/mn_lspipeline/_data",
>                 "Destination": "/usr/share/logstash/pipeline",
>                 "Driver": "local",
>                 "Mode": "rw",
>                 "RW": true,
>                 "Propagation": ""
>             }
>         ],


> bash-4.2$ mount
> overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2
> ...
> /dev/sdd6 on /usr/share/logstash/pipeline type ext4 (rw,relatime,errors=remount-ro,data=ordered)
> ...

I also have an elasticsearch container with a mounted data directory, and that works fine. Also have one with red node, working fine as well. Only logstash seems to be a problem.

I cannot figure it out. Help please! :slightly_smiling_face: Thanks!

+1 .

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.