Authentication to realm default_native failed - Password authentication failed

[natgrant (Natalie)]

Hi there,

I am hoping to see if anyone can help me with the below error occurring on our ingest nodes -

[o.e.x.s.a.AuthenticationService] [elastic-ingest-1] Authentication to realm default_native failed - Password authentication failed for auditbeat_user

I note this is an authentication failure is concerning the native realm, so I have checked that the user auditbeat_user does exist under Management / Security / Users within the Kibana UI.

I have also checked the vault to make sure the auditbeat_user has the correct password, and have executed a curl request on the ingest node to ensure the user can authenticate with that password (returns the server certificate and HTTP 200 - so I know this is all OK).

Does this indicate that one of the servers we have pointing to the ingest node with beats likely has an issue authenticating with the auditbeat_user?

I can see that data is being ingested as the 'discover' tab on the Kibana UI still shows auditbeat data. We have many platforms hooked up to this cluster so I have yet to check that all can authenticate to our ingest node (but thought there would be more information in the log if this was the case).

n.b. Elastic version 7.6.2, beats running same version.
The native realm has not been configured in the elasticsearch.yml.

xpack settings are -

    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.supported_protocols: 'TLSv1.3'
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.key: '/etc/elasticsearch/elastic-ingest-1.key'
    xpack.security.transport.ssl.certificate: '/etc/elasticsearch/elastic-ingest-1.crt'
    xpack.security.transport.ssl.certificate_authorities: 
    ['/etc/elasticsearch/elastic_transport_ssl_ca.crt' ]

    #xpack.security.transport.filter.enabled: true
    #xpack.security.transport.filter.allow: [ 'localhost' ]
    #xpack.security.transport.filter.deny: _all

    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.supported_protocols: 'TLSv1.3'
    xpack.security.http.ssl.key: '/etc/elasticsearch/elastic-ingest-1_api.key'
    xpack.security.http.ssl.certificate: '/etc/elasticsearch/elastic-ingest-1_api.crt'

    xpack.security.http.filter.enabled: true
    xpack.security.http.filter.allow: ["localhost", "0.0.0.0"]
    xpack.security.http.filter.deny: _all
    xpack.monitoring.collection.enabled: false

Thanks,
Nat

[TimV (Tim Vernum)]

@natgrant

Does this indicate that one of the servers we have pointing to the ingest node with beats likely has an issue authenticating with the auditbeat_user?

Yes, this is explicitly a password failure. It is a sign that something is trying to authenticate with an incorrect password (or no password).

You could try turning on audit logging in order to capture more detailed information about the requests that are failing (client ip, etc).