Authentication to realm active_directory failed on Elasticsearch

[lcui_dxc (Li Cui)]

Hello there,

This afternoon, when I tried to log on to Kibana with my ID Via AD as usual, I got "Oops, please try again later". Then I used the internal builtin use"elastic", I got the same.
I was able to SSH to all the ELK nodes (Linux servers on AWS) using my ID as usual via AD, no problem on Linux, but when I tried to log on any elasticsearch (ES) node with http://elasticNodexHost.com:9200 I got:

"{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user for REST request [/]","header":{"WWW-Authenticate":["Basic realm="security" charset="UTF-8"","Bearer realm="security""]}}],"type":"security_exception","reason":"unable to authenticate user for REST request [/]","header":{"WWW-Authenticate":["Basic realm="security" charset="UTF-8"","Bearer realm="security""]}},"status":401}

When I tried to run:

url -k -u user:xxx 'http://elasticNode3.hls.dxc.com:9200/_xpack/security/_authenticate?pretty'
curl: (52) Empty reply from server

from one node to anther one, or on the same ES node, I got: curl: (52) Empty reply from server

And in the ES log, I saw the following entry a lot:

[2019-02-01T21:29:06,918][WARN ][o.e.x.s.a.AuthenticationService] [elasticNode2] Authentication to realm active_directory failed - authenticate failed (Caused by LDAPException(resultCode=49 (invalid credentials), errorMessage='80090308: LdapErr: DSID-0C09042F, comment: AcceptSecurityContext error, data 52e, v2580', diagnosticMessage='80090308: LdapErr: DSID-0C09042F, comment: AcceptSecurityContext error, data 52e, v2580'))

We didn't change anything on the ES configuration at all, but it seems to me that the communication between ES nodes and AD and among the ES nodes are having issues.

This morning, my colleague was able to log on to Kibana with his ID via AD I know... something may be changed..

Here is the ES configuration with AD:

xpack:
security:
authc:
realms:
active_directory:
type: active_directory
order: 0
domain_name: hls.dxc.com
files.role_mapping: /etc/elasticsearch/role_mapping.yml
bind_dn: CN=admin,CN=Users,DC=xxx,DC=yyy,DC=com
bind_password: pass

This configuration has been on ES nodes for months and worked fine, no change.

Any this is weekend, we need to fix this before next Monday...
Any help will be highly appreciated...

Thank you very much

Li

[lcui_dxc (Li Cui)]

AD has no issue, because I'm able to log on (SSH) to any of our Instances/servers with my ID via AD.
I'm able to SSH to any of the ES nodes via AD/LDAP with my ID without issues. Normally, I use my ID to log on to Kibana via AD, now I can not use my ID, even I tried to use internal user: 'elastic', failed, but I can use the internal user 'kibana' to log on to kibana, the kibana user is local, it doesn't need to to go to AD or to any ES nodes... user elastic may need to go from kibana node to ES nodes, which is having issues I guess.

Any help would be appreciated...

Li

[ikakavas (Ioannis Kakavas)]

Hi Li,

This is a community forum, it may take time for someone to reply to your question, even more on a weekend.

If you are in need of a service with an SLA that covers response times for questions then you may want to consider talking to us about a subscription.

A few notes:

resultCode=49 (invalid credentials), errorMessage='80090308: LdapErr: DSID-0C09042F, comment: AcceptSecurityContext error, data 52e, v2580', diagnosticMessage='80090308: LdapErr: DSID-0C09042F, comment: AcceptSecurityContext error, data 52e, v2580')

This is pretty clear on the problem being invalid credentials for AD. See also AD error codes

@lcui_dxc

I'm able to log on (SSH) to any of our Instances/servers with my ID via AD

Not sure what you mean with this, but I'm guessing you are using something like libpam-ldap and your credentials are actually checked against AD ?

even I tried to use internal user: 'elastic', failed, but I can use the internal user 'kibana' to log on to kibana, the kibana user is local, it doesn't need to to go to AD or to any ES nodes... user elastic may need to go from kibana node to ES nodes, which is having issues I guess.

Kibana has no local users, both kibana and elastic are built-in users of the reserved realm in Elasticsearch. Could it be that you have a wrong password for the elastic user and a correct one for the kibana user?

I'd suggest that you

1. Look at your AD logs, that will give you a better understanding of what the error is and at least will confirm that the issue is invalid credentials

2. Enable the debug logging for AD in Elasticsearch: add the following lines to the end of the log4j2.properties configuration file in the ES_PATH_CONF :

logger.authc.name = org.elasticsearch.xpack.security.authc 
logger.authc.level = DEBUG

and share a larger part of your logs

3. Verify that your bind DN and password are correct by i.e. doing an ldapsearch with these credentials against AD

[lcui_dxc (Li Cui)]

Hello,

This was fixed... someone changed the password we used to bind to LDAP/AD, after we reset the password back to what it was, problem fixed.

Thanks a lot

Li