Got response code '401' contacting Elasticsearch at URL

[vaclav1]

Dear team,

I have enabled TLS on cluster and wanted to switch communication between logstash and Elasticsearch to secure protocol. After I set up all I received following error while starting logstash (btw. password is correct I tried it):

[2021-06-01T11:35:43,223][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_write:xxxxxx@10.194.144.63:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'https://10.194.144.63:9200/'"}

My output pipe:

elasticsearch {
        hosts => [ "https://10.194.144.64", "https://10.194.144.62", "https://10.194.144.63" ]
        cacert => "/app/products/logstash/conf/certs/elasticsearch-ca.pem"
        index => "log-beancounter"
        user => "logstash_write"
        password => "*******"
        template => "/app/products/logstash/conf/templates/log-beancounter.json"
        template_name => "log-beancounter"
        template_overwrite => "true"
        manage_template => true
        document_id => "%{[@metadata][fingerprint]}"
      }

User has following role assigned (I believe it is sufficient to do any operations):

{
  "logstash_write" : {
    "cluster" : [
      "manage_ilm",
      "manage_index_templates",
      "manage_ingest_pipelines",
      "manage_rollup",
      "read_ilm",
      "monitor",
      "manage",
      "all"
    ],
    "indices" : [
      {
        "names" : [
          "*"
        ],
        "privileges" : [
          "all"
        ],
        "allow_restricted_indices" : false
      }
    ],
    "applications" : [ ],
    "run_as" : [ ],
    "metadata" : { },
    "transient_metadata" : {
      "enabled" : true
    }
  }
}

[warkolm (Mark Walkom)]

401 means the auth details are wrong, can you try it with a curl?

[vaclav1]

Hi, I tried, curl returns the same. I tried even to change the PWD, no luck either with new password. What else might be wrong? Maybe we need api keys? WIth superuser I have no issues, or applications definitions?

[vaclav1]

Ok, I think I found an Issue, somewhere in documentation is mentioned this:

If your node has xpack.security.http.ssl.enabled set to true , then you must specify https when creating your API key.

I think the same applies for the user, now question, how to specify this? I could not find more information sadly..

[TimV (Tim Vernum)]

@vaclav1

curl returns the same.

Does that mean curl works or fails?

Can you provide the exact test that you ran?

@vaclav1

I think the same applies for the user,

There is no SSL requirement for users, and even if there was, your original logstash configuration uses https.

[vaclav1]

I found and solve the issue enabled debug mode helped me. I forgot to explicitly define in output Elasticsearch logstash plugin ssl => true