Posting this for the record, should anyone else run into the same problem...
LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError: Got response code '403' contacting Elasticsearch at URL 'https://es01:9200/logstash' perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.1-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80
I've run through a couple of new installations of Elastic Stack over the past two months, with a particular issue of not being able to get authentication for publishing to Elasticsearch from Logstash to work. Works fine with a "superuser" userid but not with any other credential that doesn't have this level of privileges.
As I said, a new install; I hadn't done any work with this stuff in a few years but I had some background; followed the guides and documentation online to get a three-cluster (container, in my case) ES cluster running, with Kibana and a Logstash container.
The reason for the background is that I specifically backed off from where I started, and was using the documentation for Configuring Security in Logstash to the point of pasting in the examples for defining a role, creating a user with that role, and creating an API key to use.
The issue seems to be this: while all of the configuration for Logstash uses the string
logstash-* as an index template - as in all of the Kibana references, the abovementioned examples, and other questions I saw posted while researching this, in my environment it causes this 403 issue, and the fix is to express the pattern as
logstash* (i.e., no hyphen). This was implied in at least one other post, but not confirmed, so I wish to confirm it.
It's also the case that although I didn't quite get it on first (and fourth) reading, creating an API key doesn't have any relationship to a user - meaning that for setting access for an API key, the same
logstash* string needs to be used to specify the index pattern for access, or else a 403 results when trying to use that API key for access.