Hi,
I had logstash stalled, I've used the --pipeline.unsafe_shutdown but no luck. Here are the logs.
ubuntu@elk:~$ sudo service logstash start --pipeline.unsafe_shutdown
^C
ubuntu@elk:~$ cd /usr/share/logstash/
ubuntu@elk:/usr/share/logstash$ bin/logstash -e 'input { generator { } } filter { ruby { code => "sleep 10000" } } output { stdout { codec => dots } }' -w 1 --pipeline.unsafe_shutdown
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[FATAL] 2020-05-21 14:39:09.858 [main] runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:528:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:288:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:199:in `block in validate_all'", "org/jruby/RubyHash.java:1415:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:198:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:305:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:263:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:88:in `<main>'"]}
[ERROR] 2020-05-21 14:39:09.863 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
ubuntu@elk:/usr/share/logstash$ sudo bin/logstash -e 'input { generator { } } filter { ruby { code => "sleep 10000" } } output { stdout { codec => dots } }' -w 1 --pipeline.unsafe_shutdown
[sudo] password for ubuntu:
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-05-21 14:39:41.348 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-05-21 14:39:41.354 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.7.0"}
[INFO ] 2020-05-21 14:39:42.460 [Converge PipelineAction::Create<main>] Reflections - Reflections took 22 ms to scan 1 urls, producing 21 keys and 41 values
[WARN ] 2020-05-21 14:39:42.683 [[main]-pipeline-manager] javapipeline - 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[WARN ] 2020-05-21 14:39:42.724 [[main]-pipeline-manager] LazyDelegatingGauge - A gauge metric of an unknown type (org.jruby.RubyArray) has been created for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[INFO ] 2020-05-21 14:39:42.726 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x409a8d48 run>"}
[INFO ] 2020-05-21 14:39:43.178 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2020-05-21 14:39:43.219 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2020-05-21 14:39:43.377 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9601}
^[[A^[[B^C[WARN ] 2020-05-21 14:57:06.758 [SIGINT handler] runner - SIGINT received. Shutting down.
^C[FATAL] 2020-05-21 14:57:06.959 [SIGINT handler] runner - SIGINT received. Terminating immediately..
[ERROR] 2020-05-21 14:57:06.989 [LogStash::Runner] Logstash - org.jruby.exceptions.ThreadKill
^Cubuntu@elk:/usr/share/logstashsudo tail -f /var/log/logstash/logstash-plain.log
[sudo] password for ubuntu:
[2020-05-21T14:56:59,271][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://192.168.2.220:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://192.168.2.220:9200/'"}
[2020-05-21T14:56:59,919][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>34, "name"=>"[main]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:34:in `sleep'"}, {"thread_id"=>35, "name"=>"[main]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:34:in `sleep'"}, {"thread_id"=>36, "name"=>"[main]>worker2", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:34:in `sleep'"}, {"thread_id"=>37, "name"=>"[main]>worker3", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:34:in `sleep'"}, {"thread_id"=>38, "name"=>"[main]>worker4", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:34:in `sleep'"}, {"thread_id"=>39, "name"=>"[main]>worker5", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:34:in `sleep'"}, {"thread_id"=>40, "name"=>"[main]>worker6", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:34:in `sleep'"}, {"thread_id"=>41, "name"=>"[main]>worker7", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:34:in `sleep'"}]}}
Any help please,
Regards,