So I've been googling this issue and trying multiple solutions but so far nothing has worked. I'm posting this here in the hopes that I've missed something else.
I have a single ES cluster running on server with Kibana on the same server.
I have several remote machines running filebeat and one remote machine running logstash. They are all running 7.2.1 and have the basic xpack enabled.
When I first enabled the basic xpack, I checked and I was able to log into Kibana no problem with my new account username password. But I forgot to modify the config files for the filebeat and LS. When I realized that I had not logs coming into ES, I went back and put the elastic username/password info into the filebeat.yml and logstash.yml files. I verified that logs were once again going into ES. But then Kibana stopped working. In the log I was getting the following errors:
"PollError" [security_exception] failed to authenticate user [elastic], with { header={ WWW-Authenticate="Basic realm="security" charset="UTF-8""} }
On restarts the errors for each of the components in the log stating the following:
Status changed from yellow to red - [security_exception] failed to authenticate user [elastic], with { header={ WWW-Authenticate="Basic realm="security" charset="UTF-8""} }
In my kibana.yml I have:
elasticsearch.username:elastic
elasticsearch.password:password
where the password is the same as in the logstash.yml and filebeat.yml and those are working. That same password also works just fine when I try: curl -XGET -u elastic 'http://localhost:9200/_cluster/health?pretty'
I tried simplifying the password in case one of the special characters was screwing things up but that didn't fix anything.
I'm running on CentOS 7.x
I've tried starting via the command line in case the systemctl configs were corrupted or something, but that didn't help. I'm wondering what else I may have changed while fixing the first things...
Suggestions?
EDIT: In my elasticsearch.yml, I have xpack.security.enabled: true
and in my kibana.yml I have the same at the bottom.
I then sniffed the traffic going from kibana to elasticsearch and saw the following:
GET /_template/.kibana_task_manager?filter_path=*.version HTTP/1.1
Authorization: Basic base64_hash
Host: elasticsearch_IP:9200
Content-Length: 0
Connection: keep-alive
as well as several other GET and POST commands all using the same base64 hash. When I decoded that hash, it matched the elastic username/password I had set.
So this just deepens the mystery of why I'm getting that error when I know that password works in Logstash, Filebeat, and with curl commands?