When using the official image for logstash (8.0.0) if a plugin is installed, after that Logstash will not load. If I use the image without the plugin installed then it works. For example:
docker run --rm --name my-logstash -v $(pwd):/usr/share/logstash/pipeline/ docker.elastic.co/logstash/logstash:8.0.0 sh -c "logstash-plugin install logstash-integration-jdbc ; /usr/local/bin/docker-entrypoint"
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Validating logstash-integration-jdbc
Resolving mixin dependencies
Updating mixin dependencies logstash-mixin-ecs_compatibility_support, logstash-mixin-validator_support, logstash-mixin-event_support
Bundler attempted to update logstash-mixin-ecs_compatibility_support but its version stayed the same
Bundler attempted to update logstash-mixin-validator_support but its version stayed the same
Bundler attempted to update logstash-mixin-event_support but its version stayed the same
Installing logstash-integration-jdbc
Installation successful
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2022-02-28T21:00:38,867][INFO ][logstash.runner ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
[2022-02-28T21:00:38,890][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.0.0", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2022-02-28T21:00:38,894][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/]
[2022-02-28T21:00:38,969][INFO ][logstash.settings ] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2022-02-28T21:00:38,991][INFO ][logstash.settings ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2022-02-28T21:00:39,637][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"d5d25d59-bb15-4bc3-88a6-f323901714b6", :path=>"/usr/share/logstash/data/uuid"}
[2022-02-28T21:00:39,738][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: wrong number of arguments (given 3, expected 1..2)>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/api/modules/base.rb:43:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.2.0/lib/sinatra/base.rb:1537:in `new'", "/usr/share/logstash/logstash-core/lib/logstash/api/rack_app.rb:113:in `block in app'", "org/jruby/RubyBasicObject.java:2622:in `instance_eval'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/builder.rb:125:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/api/rack_app.rb:101:in `app'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:123:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:74:in `from_settings'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:508:in `create_agent'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:399:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:68:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:282:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:133:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:93:in `<main>'"]}
[2022-02-28T21:00:39,749][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
If I do the same thing without the plugin line, then it works fine:
docker run --rm --name my-logstash -v $(pwd):/usr/share/logstash/pipeline/ docker.elastic.co/logstash/logstash:8.0.0
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2022-02-28T21:01:53,353][INFO ][logstash.runner ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
[2022-02-28T21:01:53,373][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.0.0", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2022-02-28T21:01:53,376][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/]
[2022-02-28T21:01:53,433][INFO ][logstash.settings ] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2022-02-28T21:01:53,453][INFO ][logstash.settings ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2022-02-28T21:01:54,208][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"ce631c8e-96a9-443e-a020-ec5bbdd822e5", :path=>"/usr/share/logstash/data/uuid"}
[2022-02-28T21:01:55,934][WARN ][logstash.monitoringextension.pipelineregisterhook] xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set `xpack.monitoring.enabled: true` in logstash.yml
[2022-02-28T21:01:55,939][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
Please configure Metricbeat to monitor Logstash. Documentation can be found at:
https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
[2022-02-28T21:01:57,481][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2022-02-28T21:01:57,885][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"elasticsearch: Name or service not known", :exception=>Manticore::ResolutionFailure, :cause=>java.net.UnknownHostException: elasticsearch: Name or service not known}
[2022-02-28T21:01:57,896][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"}
[2022-02-28T21:01:57,998][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"elasticsearch", :exception=>Manticore::ResolutionFailure, :cause=>java.net.UnknownHostException: elasticsearch}
[2022-02-28T21:01:58,009][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/_xpack][Manticore::ResolutionFailure] elasticsearch {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/_xpack][Manticore::ResolutionFailure] elasticsearch", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2022-02-28T21:01:58,022][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/_xpack][Manticore::ResolutionFailure] elasticsearch"}
[2022-02-28T21:01:58,104][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
[2022-02-28T21:01:58,498][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-02-28T21:02:00,337][INFO ][org.reflections.Reflections] Reflections took 190 ms to scan 1 urls, producing 120 keys and 417 values
[2022-02-28T21:02:01,288][INFO ][logstash.codecs.jsonlines] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2022-02-28T21:02:01,339][INFO ][logstash.javapipeline ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-02-28T21:02:01,544][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/usr/share/logstash/pipeline/01-courses.conf"], :thread=>"#<Thread:0x6274530 run>"}
[2022-02-28T21:02:02,899][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.34}
[2022-02-28T21:02:02,939][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-02-28T21:02:03,066][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{"host":{"name":"f2bdb7b37454"},"@timestamp":"2022-02-28T21:02:03.085907Z","message":"line 1","event":{"sequence":1,"original":"line 1"},"@version":"1"}
{"host":{"name":"f2bdb7b37454"},"@timestamp":"2022-02-28T21:02:03.087938Z","message":"line 2","event":{"sequence":2,"original":"line 2"},"@version":"1"}
{"host":{"name":"f2bdb7b37454"},"@timestamp":"2022-02-28T21:02:03.083608Z","message":"line 2","event":{"sequence":0,"original":"line 2"},"@version":"1"}
{"host":{"name":"f2bdb7b37454"},"@timestamp":"2022-02-28T21:02:03.086777Z","message":"line 3","event":{"sequence":1,"original":"line 3"},"@version":"1"}
{"host":{"name":"f2bdb7b37454"},"@timestamp":"2022-02-28T21:02:03.084726Z","message":"line 3","event":{"sequence":0,"original":"line 3"},"@version":"1"}
{"host":{"name":"f2bdb7b37454"},"@timestamp":"2022-02-28T21:02:03.087173Z","message":"line 1","event":{"sequence":2,"original":"line 1"},"@version":"1"}
{"host":{"name":"f2bdb7b37454"},"@timestamp":"2022-02-28T21:02:03.038693Z","message":"line 1","event":{"sequence":0,"original":"line 1"},"@version":"1"}
{"host":{"name":"f2bdb7b37454"},"@timestamp":"2022-02-28T21:02:03.086305Z","message":"line 2","event":{"sequence":1,"original":"line 2"},"@version":"1"}
{"host":{"name":"f2bdb7b37454"},"@timestamp":"2022-02-28T21:02:03.088515Z","message":"line 3","event":{"sequence":2,"original":"line 3"},"@version":"1"}
[2022-02-28T21:02:03,334][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2022-02-28T21:02:03,683][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2022-02-28T21:02:03,753][INFO ][logstash.runner ] Logstash shut down.
I also tried making a custom image from the base and it resulted in a similar error. The logstash pipleline is a simple as possible to illustrate it should not pipeline related:
input {
generator {
lines => [
"line 1",
"line 2",
"line 3"
]
# Emit all lines 3 times.
count => 3
}
}
output {
stdout { codec => json_lines }
}