Our usage of logstash requires that we run as user nobody:nogroup and we run with an appropriate securityContext in K8s that enforces it + denies PriviledgeEscalation to root
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- all
readOnlyRootFilesystem: true
runAsGroup: 65534
runAsUser: 65534
All was working fine until we upgraded to logstash:7.17.8
After investigation, we found that logstash was failing to start - it seems to be related to user security Privileges in logstash specifically and nothing to do with our configuration, file system permissions, or anything else we are doing. It looks to be specifically related to logstash:7.17.8 and we can reproduce the issue running the plain vanilla docker images with default config.
The following steps we done to isolate the issue:
- Changed to allow default securityContext - run as root - Works
- Restored securityContext - removing all of our specific logstash configuration - Still Fails
- We even attempted to run logstash using strace to see if we can isolate the system security related issue - we did not see it using strace.
- We then attempted to reproduce the problem with the specific docker images
- We found that when running logstash in the released docker images 7.17.7 vs 7.17.8 and the problem doesn't exist in 7.17.7, yet does in 7.17.8
Reproduction Steps:
# 7.17.8 Failure - using nobody:nogroup
mkdir data.7.17.8
chown nobody data.7.17.8
chmod -R ugo+rw data.7.17.8
docker run -u nobody:nogroup -v $PWD/data.7.17.8:/usr/share/logstash/data --rm -ti --entrypoint bash docker.elastic.co/logstash/logstash:7.17.8
nobody@0a35bff2d197:/usr/share/logstash$ logstash -e 'input { stdin { } } output { stdout {} }'
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[FATAL] 2023-01-10 18:30:31.009 [main] Logstash - Logstash stopped processing because of an error: (LoadError) no such file to load -- logstash/build
org.jruby.exceptions.LoadError: (LoadError) no such file to load -- logstash/build
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]
at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/api/commands/system/basicinfo_command.rb:20) ~[?:?]
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]
at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/api/command_factory.rb:19) ~[?:?]
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]
at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/api/modules/base.rb:19) ~[?:?]
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]
at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/api/rack_app.rb:20) ~[?:?]
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]
at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/webserver.rb:18) ~[?:?]
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]
at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:23) ~[?:?]
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:44) ~[?:?]
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:92) ~[?:?]
nobody@0a35bff2d197:/usr/share/logstash$ exit
Doing the same using 7.17.7 works as expected - stopped with CTRL-C
# Success with 7.17.7 - using nobody:nogroup - CTRL-C to stop
mkdir data.7.17.7
chown nobody data.7.17.7
chmod -R ugo+rw data.7.17.7
docker run -u nobody:nogroup -v $PWD/data.7.17.7:/usr/share/logstash/data --rm -ti --entrypoint bash docker.elastic.co/logstash/logstash:7.17.7
nobody@d47a3308a495:/usr/share/logstash$ logstash -e 'input { stdin { } } output { stdout {} }'
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2023-01-10T18:50:03,183][INFO ][logstash.runner ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
[2023-01-10T18:50:03,200][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.17.7", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.16+8 on 11.0.16+8 +indy +jit [linux-x86_64]"}
[2023-01-10T18:50:03,204][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[2023-01-10T18:50:03,254][INFO ][logstash.settings ] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2023-01-10T18:50:03,274][INFO ][logstash.settings ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2023-01-10T18:50:03,806][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-01-10T18:50:03,851][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"6f060474-0c58-48ab-80dd-39a196200fc1", :path=>"/usr/share/logstash/data/uuid"}
[2023-01-10T18:50:05,320][WARN ][logstash.monitoringextension.pipelineregisterhook] xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set `xpack.monitoring.enabled: true` in logstash.yml
[2023-01-10T18:50:05,323][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and may be removed in a future release.
Please configure Metricbeat to monitor Logstash. Documentation can be found at:
https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
[2023-01-10T18:50:05,746][WARN ][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2023-01-10T18:50:05,813][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2023-01-10T18:50:06,128][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2023-01-10T18:50:06,268][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"elasticsearch: Temporary failure in name resolution", :exception=>Manticore::ResolutionFailure, :cause=>java.net.UnknownHostException: elasticsearch: Temporary failure in name resolution}
[2023-01-10T18:50:06,273][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Temporary failure in name resolution"}
[2023-01-10T18:50:06,308][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"elasticsearch", :exception=>Manticore::ResolutionFailure, :cause=>java.net.UnknownHostException: elasticsearch}
[2023-01-10T18:50:06,313][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/_xpack][Manticore::ResolutionFailure] elasticsearch {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/_xpack][Manticore::ResolutionFailure] elasticsearch", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2023-01-10T18:50:06,322][WARN ][logstash.licensechecker.licensereader] Attempt to validate Elasticsearch license failed. Sleeping for 0.02 {:fail_count=>1, :exception=>"Elasticsearch Unreachable: [http://elasticsearch:9200/_xpack][Manticore::ResolutionFailure] elasticsearch"}
[2023-01-10T18:50:06,348][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
[2023-01-10T18:50:06,385][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
[2023-01-10T18:50:06,651][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
^C[2023-01-10T18:50:07,948][WARN ][logstash.runner ] SIGINT received. Shutting down.
[2023-01-10T18:50:08,748][INFO ][org.reflections.Reflections] Reflections took 158 ms to scan 1 urls, producing 119 keys and 419 values
[2023-01-10T18:50:09,867][WARN ][deprecation.logstash.codecs.line] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2023-01-10T18:50:09,902][WARN ][deprecation.logstash.inputs.stdin] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2023-01-10T18:50:10,486][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>16, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2000, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x2fd5fd2a run>"}
^C[2023-01-10T18:50:11,231][FATAL][logstash.runner ] SIGINT received. Terminating immediately..
[2023-01-10T18:50:11,382][FATAL][org.logstash.Logstash ]
org.jruby.exceptions.ThreadKill: null
nobody@d47a3308a495:/usr/share/logstash$ exit
NOTE: Using 7.18.8 with root user works with docker
# Running 7.17.8 as root works - no need to show full output
mkdir data.7.17.8.root
chmod -R ugo+rw data.7.17.8.root
docker run -v $PWD/data.7.17.8.root:/usr/share/logstash/data --rm -ti --entrypoint bash docker.elastic.co/logstash/logstash:7.17.8
logstash@d0cc92175a9e:~$ logstash -e 'input { stdin { } } output { stdout {} }'
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
...
Has Anybody else run into this issue? Is there a workaround for this?
NOTE: I noticed in the release notes for 7.17.8 that the following plugins + the JDK were updated:
- logstash-codec-cef (6.2.6) - from 6.2.5
- logstash-filter-grok (4.4.3) - from 4.4.2
- logstash-output-tcp (6.0.3) - from 6.0.2
Something in logstash is broken in 7.x
Also Created an issue on github: Regression: Run as user nobody:nogroup was broken in logstash:7.17.8 (logstash:7.17.7 works as expected)
