Logstash multiple pipeline on docker

Elastic Stack Logstash
1

Hi All
I'm trying to build logstash on docker with multiple pipelines.
After reading a lot over the internet and trying things, still can't start multiple pipelines on docker.
I'm trying to run some basic configuration in order to ensure that the mechanism is working , I see that logstash is running , but can't see multiple piplines are working

pipelines.yml looks like this :

- pipeline.id: A11111111

config.string: |
               input {
              generator {
                lines => [
                  "line 1",
                  "line 2",
                  "line 3"
                ]
                count => 1
              }
            }
        output {
           stdout{id => "stdout_plugin111"}
        }

- pipeline.id: B22222222

config.string: |
               input {
              generator {
                lines => [
                  "line 4",
                  "line 5",
                  "line 6"
                ]
                count => 1
              }
            }
        output {
           stdout{id => "stdout_plugin222"}
        }

starting docker from cmd:

> docker run --rm -it -v /c/logstash/:/usr/share/logstash/pipelines docker.elastic.co/logstash/logstash:7.8.0

when in "c:\logstash" directory, pipelines.yml file located.

Output looks like this

image
image1977×804 84.8 KB

Thanks in advance for your help.

2

Clearly it is not loading the pipelines.yml that you want it to load. logstash will load pipelines.yml from path.settings

Try enabling log.level debug and see what path.settings is set to.

3

[2020-07-16T07:49:39,624][DEBUG][logstash.runner ] path.settings: "/usr/share/logstash/config"

this is a path.settings, is it OK ?

4

Is that where pipelines.yml exists? If so, does the use who is running logstash have permissions to the file and directory? Also, if this is docker, are you sure the file exists in the container?

5

@Badger thanks for reply.
After I placed logstash,yml and pipelines.yml to the same directory (c:\Logstash in my case) and set path config of the second pipeline (as additional file) it worked.

#pipelines.yml

- pipeline.id: test1111111111111
  pipeline.workers: 1
  pipeline.batch.size: 1
  config.string: 'input { 
  generator {  
  lines => [  "line 4",   "line 5",   "line 6"   ]     
  count => 1      }    } 
  output {   stdout{id => "stdout_plugi3333322"}}'

- pipeline.id: test2222222222222
  pipeline.workers: 1
  path.config: "/usr/share/logstash/config/second.conf"

#second.conf in the same directory as pipelines.yml and logstash.yml
input {
generator {
lines => [ "line 4", "line 5", "line 6" ]
count => 1 } }
output { stdout{id => "stdout_plugin22222"}}

command line run of logstash in docker

docker run --rm -it -v /c/logstash/:/usr/share/logstash/config docker.elastic.co/logstash/logstash:7.8.0 --log.level=debug

  • This is a working example for those who looking to run multiple pipelines of logstash on docker.