Hi, thanks for getting back to me.
I've tested the upgrade but few issues appeared:
- ElasticSearch output plugin doesn't add anything into [tags] even after it fails to index (all nodes are OFF). How to know if the index worked or failed? here's what I did:
output {
if ![tags] {
elasticsearch {
hosts => "${ES_HOSTS}"
doc_as_upsert => true
document_id => "%{[@metadata][fingerprint]}"
index => "teste"
}
}
if [tags] {
stdout { } # never prints even though ES its OFF
}
}
- It's impossible to stop Logstash after the attempt to index one message into Elasticsearch, I had to kill it, see below.
[2018-05-25T11:34:03,508][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-05-25T11:34:03,512][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused"}
^C[2018-05-25T11:34:05,591][WARN ][logstash.runner ] SIGINT received. Shutting down.
[2018-05-25T11:34:08,527][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-05-25T11:34:08,530][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused"}
[2018-05-25T11:34:10,593][WARN ][logstash.runner ] Received shutdown signal, but pipeline is still waiting for in-flight events
to be processed. Sending another ^C will force quit Logstash, but this may cause
data loss.
[2018-05-25T11:34:10,614][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>8, "stalling_thread_info"=>{["LogStash::Filters::Mutate", {"remove_field"=>["message"], "id"=>"3dca6f301a96874f91e44fc656cfc92b03a1f4a5055fb8d4479c5ce9ddbaed2e"}]=>[{"thread_id"=>25, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.4-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}, {"thread_id"=>26, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.4-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}, {"thread_id"=>27, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.4-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}, {"thread_id"=>28, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.4-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}, {"thread_id"=>29, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.4-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}, {"thread_id"=>30, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.4-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}, {"thread_id"=>31, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.4-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}, {"thread_id"=>32, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.4-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}]}}
[2018-05-25T11:34:10,615][ERROR][logstash.shutdownwatcher ] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
[2018-05-25T11:34:13,551][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2018-05-25T11:34:13,554][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused"}
1[2018-05-25T11:34:15,616][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>8, "stalling_thread_info"=>{["LogStash::Filters::Mutate", {"remove_field"=>["message"], "id"=>"3dca6f301a96874f91e44fc656cfc92b03a1f4a5055fb8d4479c5ce9ddbaed2e"}]=>[{"thread_id"=>25, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.4-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}, {"thread_id"=>26, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.4-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}, {"thread_id"=>27, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.4-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}, {"thread_id"=>28, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.4-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}, {"thread_id"=>29, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.4-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}, {"thread_id"=>30, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.4-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}, {"thread_id"=>31, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.4-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}, {"thread_id"=>32, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.4-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `sleep'"}]}}
^C[2018-05-25T11:34:17,039][FATAL][logstash.runner ] SIGINT received. Terminating immediately..
[2018-05-25T11:34:17,173][ERROR][org.logstash.Logstash ] org.jruby.exceptions.ThreadKill
Thanks.