Hello! I am facing difficulties trying to configure my ELK stack in which Logstash should be running multiple pipelines, and the problem seems to be in the Logstash docker-compose.yml configuration. In fact, I get the following error whenever I try to run it:
88755932b36c_logstash | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
88755932b36c_logstash | [2021-03-06T15:23:55,465][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.11.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [linux-x86_64]"}
88755932b36c_logstash | ERROR: Failed to read pipelines yaml file. Location: /usr/share/logstash/config/pipelines.yml
88755932b36c_logstash | usage:
88755932b36c_logstash | bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
88755932b36c_logstash | bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
88755932b36c_logstash | bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
88755932b36c_logstash | bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
88755932b36c_logstash | bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
88755932b36c_logstash | bin/logstash --help
88755932b36c_logstash | [2021-03-06T15:23:55,975][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
88755932b36c_logstash | org.jruby.exceptions.SystemExit: (SystemExit) exit
88755932b36c_logstash | at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
88755932b36c_logstash | at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
88755932b36c_logstash | at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]
88755932b36c_logstash exited with code 1
I tried to follow the documentation's instructions and I read about very similar topics, but I still do not get how to fix this issue. The documentation says the configurations of the single pipelines' absolute paths (relatively to the container) must must be specified in pipelines.yml
, and /usr/share/logstash/config/
should be the path.settings
the documentation talks about. That directory (in the container) should contain both the single pipelines' .conf and the pipelines.yml
, and I think I have declared it in "volumes".
Here are the docker-compose.yml (I will only paste the Logstash Docker configuration, as Elasticsearch and Kibana work fine) and the pipelines.yml. docker-compose.yml, pipelines.yml and the two .conf files all are in the same directory and they are bound to the path.settings
. Needless to say, if I uncomment the #single pipeline
line and comment the other two bonds in "volumes", the dockerized Logstash works perfectly with a single main pipeline.
Here is the docker-compose.yml (limited to Logstash's configuration):
logstash:
image: docker.elastic.co/logstash/logstash-oss:7.11.1
container_name: logstash
hostname: logstash
volumes:
# - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf # single pipeline
- ./logstash-to-opendistro.conf:/usr/share/logstash/config/logstash-to-opendistro.conf
- ./opendistro-to-logstash.conf:/usr/share/logstash/config/opendistro-to-logstash.conf
- ./pipelines.yml:/usr/share/logstash/config/pipelines.yml
links:
- opendistro:opendistro
depends_on:
- opendistro
networks:
- elknetwork
And this is the pipelines.yml
file:
- pipeline.id: logstash-to-opendistro
path.config: "/usr/share/logstash/config/logstash-to-opendistro.conf"
pipeline.workers: 1
- pipeline.id: opendistro-to-logstash
path.config: "/usr/share/logstash/config/opendistro-to-logstash.conf"
pipeline.workers: 1
The trivial question is: how do I get my setup to work, provided that Opendistro and Kibana work just fine? More specifically: how do I bind the "physical" (i.e. non-containerized) multiple pipeline configuration files and pipelines.yml
to the Docker container's filesystem?
Thank you all in advance!
PS: please let me know if you need to see the remaining configuration files - I omitted them as I thought these were the most useful.