We just experienced an issue in Logstash after a change of the Pipeline out of Kibana in Version 6.0rc-2.
We use full X-Pack Security options and therefore had TLS encryption enabled.
In Kibana Pipeline, we unintentionally changed the credentials for Logstash authentication to a user which did not have the appropriate right to generate new indices in Elasticsearch. Logstash then was able to detect the change of the pipline, but failed with an 403 error:
[2017-11-08T17:04:37,682][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>403, :url=>"https://elasticsearch.elkxpack.svc:9200/_bulk"}
[2017-11-08T17:04:37,881][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>403, :url=>"https://elasticsearch.elkxpack.svc:9200/_bulk"}
[2017-11-08T17:04:37,979][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>403, :url=>"https://elasticsearch.elkxpack.svc:9200/_bulk"}
We then changed back to the first-used credentials. Logstash again receives the change but repeatedly gives following warning:
[2017-11-08T17:04:52,265][INFO ][logstash.pipelineaction.reload] Reloading pipeline {"pipeline.id"=>:generator}
[2017-11-08T17:04:57,276][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>179, "stalling_thread_info"=>{"other"=>[{"thread_id"=>42, "name"=>"[generator]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}, {"thread_id"=>43, "name"=>"[generator]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}, {"thread_id"=>44, "name"=>"[generator]>worker2", "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}, {"thread_id"=>45, "name"=>"[generator]>worker3", "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
We had to restart Logstash to effectively make the roll back of the credential change work again.
Greetings