Unable to authenticate user for REST request

[bhavikdhoot (Bhavik)]

Hi,

I'm trying to connect to Elasticsearch through FileBeats using the following config in my filebeat.yml.

output.elasticsearch:

Array of hosts to connect to.

#hosts: ["localhost:9200"]
hosts: ["server1:9600","server2:9600","server3:9600"]
username: "4399xxxx"
password: "abcdefgh"
protocol: "https"
ssl.certificate_authorities: "/goldeneye-ca.crt"
ssl.certificate: "/goldeneye.uk.hsbc-elk.crt"
ssl.key: "/goldeneye.uk.hsbc-elk.key"
index: "engg-%{+yyyy.MM.dd}"

But, while running FileBeat, getting the following error -

2020-09-18T19:54:29.972+0530 ERROR [publisher_pipeline_output] pipeline/output.go:155 Failed to connect to backoff(elasticsearch(https://server1:9600)): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [4399xxxx] for REST request [/]","header":{"WWW-Authenticate":["Bearer realm="security"","ApiKey","Basic realm="security" charset="UTF-8""]}}],"type":"security_exception","reason":"unable to authenticate user [4399xxxx] for REST request [/]","header":{"WWW-Authenticate":["Bearer realm="security"","ApiKey","Basic realm="security" charset="UTF-8""]}},"status":401}
2020-09-18T19:54:29.972+0530 INFO [publisher_pipeline_output] pipeline/output.go:146 Attempting to reconnect to backoff(elasticsearch(https://server1:9600)) with 1 reconnect attempt(s)
2020-09-18T19:54:29.972+0530 INFO [publisher] pipeline/retry.go:221 retryer: send unwait signal to consumer
2020-09-18T19:54:29.974+0530 INFO [publisher] pipeline/retry.go:225 done

Appreciate any help regarding this.

Thanks.

[warkolm (Mark Walkom)]

Welcome to our community!

Can you confirm that the username and password work?

[bhavikdhoot (Bhavik)]

Hello,

Yes, the username/pwd works fine while using the same with 'curl' command or else through web browser too.

[bhavikdhoot (Bhavik)]

Please let me know if anyone has any clue regarding this issue.

[bhavikdhoot (Bhavik)]

I was able to solve this error. In the Elasticsearch logs, it was throwing invalid credentials error where as in the filebeat output it shows the above error.
Digging further, we found out that there is a bug in Elasticsearch wherein if there is one or more '$' in ur password then it will throw 'invalid credentials' error in Elasticsearch.

I changed my password without any '$' and it progressed further. I hope ELK team will take a note of it and fix this bug in next releases.
Thanks.