[6.2.2] cannot connect elastic search secured cluster

[dao (olivier hodac)]

Hello I have been stucked for a while on this.

I have an elasticsearch cluster with x-pack enabled. I have added encryption using this tutorial: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/configuring-tls.html#tls-http

I have copied my elastic-stack-ca.p12 to the logstash host (which is not a cluster node). Now I want to push some data. here is my setup:

output {
	stdout { codec => dots }
  elasticsearch {
    id                => "wilco-cluster"
    hosts             =>["111.222.333.444:9200"]
    user              => "logstash_system"
    password          => "logstash_t0ps3cr3t"
    keystore          => "/etc/logstash/certs/elastic-stack-ca.p12"
    keystore_password => "t0ps3cr3t"
    doc_as_upsert     => true
    index             => "%{[@metadata][index]}"
    action            => "update"
    document_id       => "%{[@metadata][doc_id]}"
  }
}

I try my connection like this, which seems to be fine

curl -u logstash_system:logstash_t0ps3cr3t 111.222.333.444:9200
{
  "name" : "wilco-2",
  "cluster_name" : "wilco-lake",
  "cluster_uuid" : "UTIZbUvoTtizEv91Q260jQ",
  "version" : {
    "number" : "6.2.2",
    "build_hash" : "10b1edd",
    "build_date" : "2018-02-16T19:01:30.685723Z",
    "build_snapshot" : false,
    "lucene_version" : "7.2.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

Now, I run the logstash process. I have got this error in the logs of logstash, and nothing pushed in ES:

[2018-03-22T13:19:25,396][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Got response code '403' contacting Elasticsearch at URL 'http://163.172.51.40:9200/_template/logstash'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:290:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:372:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:276:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:284:in `block in head'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:331:in `template_exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:78:in `template_install'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/template_manager.rb:21:in `install'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/template_manager.rb:9:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:57:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:26:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:42:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:in `register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:352:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:352:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:735:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:362:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:289:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:249:in `block in start'"]}

in the elastic search log, I have this:

[2018-03-22T14:28:04,652] [transport] [access_denied]	origin_type=[rest], origin_address=[212.47.242.213], principal=[logstash_system], roles=[logstash_system], action=[indices:data/write/bulk], request=[BulkRequest]

what's wrong with me???

[dao (olivier hodac)]

Found the reason why: the logstash_system user is not suitable for writing into indices.

That is not really clear in the doc. Therefore, what's this user dedicated to?

[TimV (Tim Vernum)]

From

Setting Up User Authentication | X-Pack for the Elastic Stack [6.2] | Elastic

logstash_system
The user Logstash uses when storing monitoring information in Elasticsearch.

The logstash_system user is used internally within Logstash when monitoring is enabled for Logstash.

You want to look at Configuring Security in Logstash | Logstash Reference [6.2] | Elastic for setting up your own logstash user.