Logstash communication fails to ES with x-pack with user logstash_internal

[dsmteam]

I'm trying to setup latest ELK stack with x-pack but i'm failing on the last step which is communication between logstash and ES

I have put username/password in my output.conf file for logstash and it's properly created on ES with Kibana GUI. Role is setup as per documentation https://www.elastic.co/guide/en/logstash/7.1/ls-security.html#ls-monitoring-user

Everything is working fine if I use the default master "elastic" username but i'm getting the cryptic error as show below when using the logstash_writer user.
I can confirm it's not a password issue as i'm getting a 401 error when I put a wrong password in my output.conf file

This is my conf

output {
elasticsearch {
hosts => ["http://localhost:9200"]
user => ["logstash_internal"]
password => ["somepassword"]
manage_template => ["false"]
template_name => ["logstash"] }
}

I'm not sure what i'm doing wrong here...

[2019-06-04T14:17:49,901][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError: LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291:inperform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278:in block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373:inwith_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277:in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:inblock in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:341:in exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:359:inrollover_alias_exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:91:in maybe_create_rollover_alias'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:10:insetup_ilm'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:52:in `block in setup_after_successful_connection'"]}
[2019-06-04T14:17:50,097][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

[pastechecker (Charlie)]

input {}
filter {}

output {
elasticsearch {
hosts => ["http://localhost:9200"]
user => "logstash_internal"
password => "somepassword"
manage_template => "false"
template_name => "logstash"
}
}

Configuration OK
[INFO ] 2019-06-04 15:54:47.199 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

Can you remove the template_name and try again?

Your problem looks similar to the: Failed to install template logstash error

[dsmteam]

After removing template_name => same error
Also no change without manage_template...