Unable to authenticate user [elastic] for REST request [/]

[alexus]

Hello,

I'm trying to configure logs for my Elasticsearch cluster, by following this:

Elasticsearch module | Filebeat Reference [8.11] | Elastic

Configure SSL | Filebeat Reference [8.11] | Elastic

and even though i set verification_mode to none, i still getting 401

{"log.level":"error","@timestamp":"2023-09-19T19:13:02.623Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":148},"message":"Failed to connect to backoff(elasticsearch(https://es02:9200)): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}","service.name":"filebeat","ecs.version":"1.6.0"}

I tried both of these methods, however without any luck(

root@dos:/opt/elk# grep -A2 ' filebeat:' docker-compose.override.yml
  filebeat:
    command: "--strict.perms=false -e -E output.elasticsearch.ssl.verification_mode=none -E 'output.elasticsearch.hosts=[https://es01:9200,https://es02:9200,https://es03:9200]' -E output.elasticsearch.username=elastic -E output.elasticsearch.password=${ELASTIC_PASSWORD}"
#    command: "--strict.perms=false -e -E output.elasticsearch.ssl.verification_mode=certificate -E 'output.elasticsearch.ssl.certificate_authorities=[/usr/share/elk/config/certs/ca/ca.crt]' -E 'output.elasticsearch.hosts=[https://es01:9200,https://es02:9200,https://es03:9200]' -E output.elasticsearch.username=elastic -E output.elasticsearch.password=${ELASTIC_PASSWORD}"
root@dos:/opt/elk#

my environment:

root@dos:/opt/elk# uname -a
Linux X.Y.Z 6.1.0-12-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.52-1 (2023-09-07) x86_64 GNU/Linux
root@dos:/opt/elk# cat /etc/debian_version
12.1
root@dos:/opt/elk# docker version
Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:32:10 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:32:10 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
root@dos:/opt/elk#

Please advise.

[alexus]

root@a737a3ec940b:/usr/share/filebeat# filebeat --strict.perms=false -E output.elasticsearch.ssl.verification_mode=certificate -E 'output.elasticsearch.ssl.certificate_authorities=/usr/share/filebeat/config/certs/ca/ca.crt' -E 'output.elasticsearch.hosts=[https://es01:9200,https://es02:9200,https://es03:9200]' -E output.elasticsearch.username=elastic -E output.elasticsearch.password=${ELASTIC_PASSWORD} test output
elasticsearch: https://es01:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.25.0.6
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 8.10.1
elasticsearch: https://es02:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.25.0.9
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... ERROR 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
root@a737a3ec940b:/usr/share/filebeat#
root@a737a3ec940b:/usr/share/filebeat# curl --cacert /usr/share/filebeat/config/certs/ca/ca.crt --head https://elastic:$ELASTIC_PASSWORD@es01:9200
HTTP/1.1 200 OK
X-elastic-product: Elasticsearch
content-type: application/json
content-length: 523

root@a737a3ec940b:/usr/share/filebeat#

a clue maybe?