Got response code '403' contacting Elasticsearch at URL

[alexus]

Hello World!

I'm trying to follow Configuring Security in Logstash | Logstash Reference [7.17] | Elastic, specifically these:

• Configuring Logstash to use Basic Authentication
• Granting Users Access to the Logstash Indices

yet when I start Logstash instance, I get the following error:

logstash | [2023-01-17T21:58:00,448][INFO ][logstash.outputs.elasticsearch][gelf] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://elasticsearch:9200"]}
logstash | [2023-01-17T21:58:00,497][INFO ][logstash.outputs.elasticsearch][gelf] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://logstash_user:xxxxxx@elasticsearch:9200/]}}
logstash | [2023-01-17T21:58:00,669][WARN ][logstash.outputs.elasticsearch][gelf] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://logstash_user:xxxxxx@elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '403' contacting Elasticsearch at URL 'http://elasticsearch:9200/'"}

the actual username is working fine:

% curl --silent --request GET "http://logstash_user:XYZ@elasticsearch:9200/_security/_authenticate?pretty"
{
  "username" : "logstash_user",
  "roles" : [
    "logstash_reader",
    "logstash_admin"
  ],
  "full_name" : "",
  "email" : "",
  "metadata" : { },
  "enabled" : true,
  "authentication_realm" : {
    "name" : "basic1",
    "type" : "native"
  },
  "lookup_realm" : {
    "name" : "basic1",
    "type" : "native"
  },
  "authentication_type" : "realm"
}
%

Please advise.

[warkolm (Mark Walkom)]

It'd be useful if you could show us your Logstash config

[alexus]

you're absolutely right! and i should have included that with my initial question)

# docker exec -it logstash bash
logstash@b84ec064ef9a:~$ cat config/logstash.yml
http.host: 0.0.0.0
node.name: logstash
xpack.management.elasticsearch.hosts:
- http://elasticsearch:9200
xpack.management.elasticsearch.password: changeme
xpack.management.elasticsearch.username: elastic
xpack.management.enabled: true
xpack.management.pipeline.id:
- gelf
xpack.monitoring.elasticsearch.hosts:
- http://elasticsearch:9200
xpack.monitoring.elasticsearch.password: changeme
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.enabled: false
logstash@b84ec064ef9a:~$

[warkolm (Mark Walkom)]

What about the config file where you are using those credentials?

[alexus]

actually... i believe i have found an actual issue) and that is: i have granted logstash_user only logstash_reader role and not logstash_writer, and as soon as i added logstash_writer role, the error went away)

asking me about the config file where i have those credentials)))