Hello there,
This afternoon, when I tried to log on to Kibana with my ID Via AD as usual, I got "Oops, please try again later". Then I used the internal builtin use"elastic", I got the same.
I was able to SSH to all the ELK nodes (Linux servers on AWS) using my ID as usual via AD, no problem on Linux, but when I tried to log on any elasticsearch (ES) node with http://elasticNodexHost.com:9200 I got:
"{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user for REST request [/]","header":{"WWW-Authenticate":["Basic realm="security" charset="UTF-8"","Bearer realm="security""]}}],"type":"security_exception","reason":"unable to authenticate user for REST request [/]","header":{"WWW-Authenticate":["Basic realm="security" charset="UTF-8"","Bearer realm="security""]}},"status":401}
When I tried to run:
url -k -u user:xxx 'http://elasticNode3.hls.dxc.com:9200/_xpack/security/_authenticate?pretty'
curl: (52) Empty reply from server
from one node to anther one, or on the same ES node, I got: curl: (52) Empty reply from server
And in the ES log, I saw the following entry a lot:
[2019-02-01T21:29:06,918][WARN ][o.e.x.s.a.AuthenticationService] [elasticNode2] Authentication to realm active_directory failed - authenticate failed (Caused by LDAPException(resultCode=49 (invalid credentials), errorMessage='80090308: LdapErr: DSID-0C09042F, comment: AcceptSecurityContext error, data 52e, v2580', diagnosticMessage='80090308: LdapErr: DSID-0C09042F, comment: AcceptSecurityContext error, data 52e, v2580'))
We didn't change anything on the ES configuration at all, but it seems to me that the communication between ES nodes and AD and among the ES nodes are having issues.
This morning, my colleague was able to log on to Kibana with his ID via AD I know... something may be changed..
Here is the ES configuration with AD:
xpack:
security:
authc:
realms:
active_directory:
type: active_directory
order: 0
domain_name: hls.dxc.com
files.role_mapping: /etc/elasticsearch/role_mapping.yml
bind_dn: CN=admin,CN=Users,DC=xxx,DC=yyy,DC=com
bind_password: pass
This configuration has been on ES nodes for months and worked fine, no change.
Any this is weekend, we need to fix this before next Monday...
Any help will be highly appreciated...
Thank you very much
Li