This could be an issue that crosses with Grafana, but I'm hoping for some insight here. I am able to access the Elasticsearch API using my active token. I can test using curl in various python scripts to pull information and all is good.
Now I'm trying to do some log annotations in our Prometheus/Grafana environment and I'd like to simply pull some log lines from Elasticsearch. I use the Elasticsearch data source and I set our index name, and pass the 'Authorization' header with my token as 'apiKey dfjahsdfi2389yufwhuh3928fh...', etc. I get a failure to connect with the error:
Elasticsearch error: missing authentication credentials for REST request [/logstash-filebeat-2020.12.11/_mapping
Why can I use curl with my api token just fine, but cannot use the api token to connect via Grafana's Elasticsearch data source? Is there another HTTP header I need to pass that I'm not currently?
The only debug log entry I can find in Grafana is:
t=2020-12-11T10:17:41-0600 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/18/logstash-filebeat-2020.12.11/_mapping status=401 remote_addr=XXX.XXX.XX.X time_ms=7 size=241 referer=http://grafanaserver.this.domain:3000/datasources/edit/18/
Apologies if this is outside of Elasticsearch, but since my API token is active, I cant imagine why there's an issue.