LDAP Issue

Elastic Stack Kibana
1

Hi,

We have integrated Active Directory with Elastic and it is working fine and I am able to login to Kibana with my AD user (rb****).
But recently we have seen the issues with elasticsearch , elasticsearch is getting shut down frequently and the below is the WARNING we found in the logs ..

[2022-06-23T22:16:59,564][DEBUG][o.e.x.s.a.l.s.LdapUtils ] [elastic.coordinate.com] LDAP user bind [SimpleBindRequest(dn='rmc_admin@ldapserver.com', controls={AuthorizationIdentityRequestControl(isCritical=false)})] failed for [LDAPConnection(connected to ldapserver.com:389)] - [LDAPException(resultCode=49 (invalid credentials), diagnosticMessage='80090308: LdapErr: DSID-0C090447, comment: AcceptSecurityContext error, data 52e, v3839^@', ldapSDKVersion=4.0.8, revision=28812)]
[2022-06-23T22:16:59,565][DEBUG][o.e.x.s.a.l.LdapRealm ] [elastic.coordinate.com] Exception occurred during authenticate for active_directory/my_ad
com.unboundid.ldap.sdk.LDAPBindException: 80090308: LdapErr: DSID-0C090447, comment: AcceptSecurityContext error, data 52e, v3839^@
at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:2273) ~[unboundid-ldapsdk-4.0.8.jar:4.0.8]

We don't have any connection issues with LDAP and ldap users are able to login. But we are experiencing this issues frequently. Please advice

2

Those are DEBUG logs, not WARNING, do you have any WARNING or ERROR logs related to LDAP?

I'm not sure this is an issue as those logs are appearing when you have DEBUG enabled, which normally is only set if you need to troubleshoot something.

But since you are using the LDAP integration, this means that you have a license, maybe opening a ticket with support will give you a faster answer.

3

[2022-06-21T01:16:49,247][WARN ][o.e.x.s.a.RealmsAuthenticator] [elastic.coordinate.com] Authentication to realm my_ad failed - authenticate failed (Caused by LDAPException(resultCode=49 (invalid credentials), diagnosticMessage='80090308: LdapErr: DSID-0C090447, comment: AcceptSecurityContext error, data 52e, v3839^@', ldapSDKVersion=4.0.8, revision=28812))

This is the warning i got when i did not enable DEBUG.. and after this warning elastic instances went down ... so I enabled debug log to see if we get any additional info on this warning ..

event after enabling the debug, I am getting the same LDAP exception and elastic instances are getting shut down

4

This is not an arbitrary LDAP exception, it’s a message that tells you that one of your users failed to authenticate because they provided the wrong password. This is not was causing your issues and I would look for other potential culprits as to why your Elastisearch nodes shutdown. All information should be in the logs , what are the latest messages before the node stops ?

5

Hi,

when the elastic instance went down, below is the logs found in elasticsearch log file

[2022-06-29T22:10:44,089][DEBUG][o.e.x.s.a.RealmsAuthenticator] [elastic.coordinate.com] Authentication of [elastic] using realm [reserved/reserved] with token [UsernamePasswordToken] was [AuthenticationResult{status=SUCCESS, user=User[username=elastic,roles=[superuser],fullName=null,email=null,metadata={_reserved=true}], message=null, exception=null}]
[2022-06-29T22:10:44,094][DEBUG][o.e.x.s.a.e.ReservedRealm] [elastic.coordinate.com] realm [reserved] authenticated user [elastic], with roles [[superuser]] (cached)
[2022-06-29T22:10:44,094][DEBUG][o.e.x.s.a.RealmsAuthenticator] [elastic.coordinate.com] Authentication of [elastic] using realm [reserved/reserved] with token [UsernamePasswordToken] was [AuthenticationResult{status=SUCCESS, user=User[username=elastic,roles=[superuser],fullName=null,email=null,metadata={_reserved=true}], message=null, exception=null}]
[2022-06-29T22:10:44,094][DEBUG][o.e.x.s.a.e.ReservedRealm] [elastic.coordinate.com] realm [reserved] authenticated user [elastic], with roles [[superuser]] (cached)
[2022-06-29T22:10:44,094][DEBUG][o.e.x.s.a.RealmsAuthenticator] [elastic.coordinate.com] Authentication of [elastic] using realm [reserved/reserved] with token [UsernamePasswordToken] was [AuthenticationResult{status=SUCCESS, user=User[username=elastic,roles=[superuser],fullName=null,email=null,metadata={_reserved=true}], message=null, exception=null}]
[2022-06-29T22:10:44,146][DEBUG][o.e.x.s.a.e.ReservedRealm] [elastic.coordinate.com] realm [reserved] authenticated user [elastic], with roles [[superuser]] (cached)
[2022-06-29T22:10:44,146][DEBUG][o.e.x.s.a.RealmsAuthenticator] [elastic.coordinate.com] Authentication of [elastic] using realm [reserved/reserved] with token [UsernamePasswordToken] was [AuthenticationResult{status=SUCCESS, user=User[username=elastic,roles=[superuser],fullName=null,email=null,metadata={_reserved=true}], message=null, exception=null}]
[2022-06-29T22:10:44,899][DEBUG][o.e.x.s.a.e.ReservedRealm] [elastic.coordinate.com] realm [reserved] authenticated user [elastic], with roles [[superuser]] (cached)
[2022-06-29T22:10:44,899][DEBUG][o.e.x.s.a.RealmsAuthenticator] [elastic.coordinate.com] Authentication of [elastic] using realm [reserved/reserved] with token [UsernamePasswordToken] was [AuthenticationResult{status=SUCCESS, user=User[username=elastic,roles=[superuser],fullName=null,email=null,metadata={_reserved=true}], message=null, exception=null}]
[2022-06-29T22:10:46,034][DEBUG][o.e.x.s.a.RealmsAuthenticator] [elastic.coordinate.com] Authentication of [rmc_admin] using realm [reserved/reserved] with token [UsernamePasswordToken] was [AuthenticationResult{status=CONTINUE, user=null, message=null, exception=null}]
[2022-06-29T22:10:46,037][DEBUG][o.e.x.s.a.l.s.LdapUtils ] [elastic.coordinate.com] LDAP user bind [SimpleBindRequest(dn='rmc_admin@ldapserver', controls={AuthorizationIdentityRequestControl(isCritical=false)})] failed for [LDAPConnection(connected to ldapserver:389)] - [LDAPException(resultCode=49 (invalid credentials), diagnosticMessage='80090308: LdapErr: DSID-0C090447, comment: AcceptSecurityContext error, data 52e, v3839 ', ldapSDKVersion=4.0.8, revision=28812)]
[2022-06-29T22:10:46,038][DEBUG][o.e.x.s.a.l.LdapRealm ] [elastic.coordinate.com] Exception occurred during authenticate for active_directory/my_ad
com.unboundid.ldap.sdk.LDAPBindException: 80090308: LdapErr: DSID-0C090447, comment: AcceptSecurityContext error, data 52e, v3839
at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:2273) ~[unboundid-ldapsdk-4.0.8.jar:4.0.8]
at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils$2.lambda$doRun$0(LdapUtils.java:195) ~[x-pack-security-7.16.3.jar:7.16.3]
at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils.privilegedConnect(LdapUtils.java:76) ~[x-pack-security-7.16.3.jar:7.16.3]
at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils$2.doRun(LdapUtils.java:195) [x-pack-security-7.16.3.jar:7.16.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils.maybeForkAndRun(LdapUtils.java:102) [x-pack-security-7.16.3.jar:7.16.3]
at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils.maybeForkThenBind(LdapUtils.java:212) [x-pack-security-7.16.3.jar:7.16.3]
at org.elasticsearch.xpack.security.authc.ldap.ActiveDirectorySessionFactory$ADAuthenticator.authenticate(ActiveDirectorySessionFactory.java:320) [x-pack-security-7.16.3.jar:7.16.3]
at org.elasticsearch.xpack.security.authc.ldap.ActiveDirectorySessionFactory.getSessionWithoutPool(ActiveDirectorySessionFactory.java:161) [x-pack-security-7.16.3.jar:7.16.3]
at org.elasticsearch.xpack.security.authc.ldap.PoolingSessionFactory.session(PoolingSessionFactory.java:110) [x-pack-security-7.16.3.jar:7.16.3]
at org.elasticsearch.xpack.security.authc.ldap.LdapRealm.lambda$doAuthenticate$1(LdapRealm.java:147) [x-pack-security-7.16.3.jar:7.16.3]
at org.elasticsearch.xpack.security.authc.ldap.LdapRealm$CancellableLdapRunnable.doRun(LdapRealm.java:343) [x-pack-security-7.16.3.jar:7.16.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.16.3.jar:7.16.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.16.3.jar:7.16.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-06-29T22:10:46,039][DEBUG][o.e.x.s.a.RealmsAuthenticator] [elastic.coordinate.com] Authentication of [rmc_admin] using realm [active_directory/my_ad] with token [UsernamePasswordToken] was [AuthenticationResult{status=CONTINUE, user=null, message=authenticate failed, exception=LDAPException(resultCode=49 (invalid credentials), diagnosticMessage='80090308: LdapErr: DSID-0C090447, comment: AcceptSecurityContext error, data 52e, v3839 ', ldapSDKVersion=4.0.8, revision=28812)}]
[2022-06-29T22:10:46,039][WARN ][o.e.x.s.a.RealmsAuthenticator] [elastic.coordinate.com] Authentication to realm my_ad failed - authenticate failed (Caused by LDAPException(resultCode=49 (invalid credentials), diagnosticMessage='80090308: LdapErr: DSID-0C090447, comment: AcceptSecurityContext error, data 52e, v3839 ', ldapSDKVersion=4.0.8, revision=28812))
[2022-06-29T22:10:46,143][INFO ][o.e.t.ClusterConnectionManager] [elastic.coordinate.com] transport connection to [{elastic.node1}{eKXQW3zMSp-QIb9KCQz8cQ}{sby4A-WxR1KUnr3kzZ-Z6A}{elastic.node1}{192.168.1.23:9300}{his}] closed by remote
[2022-06-29T22:10:46,193][INFO ][o.e.c.s.ClusterApplierService] [elastic.coordinate.com] removed {{elastic.node1}{eKXQW3zMSp-QIb9KCQz8cQ}{sby4A-WxR1KUnr3kzZ-Z6A}{elastic.node1}{192.168.1.23:9300}{his}}, term: 29, version: 6221, reason: ApplyCommitRequest{term=29, version=6221, sourceNode={elastic.node2}{pHcFS0YKQTmhb0pWXBpz7A}{Ezc10TLbQua9ZYyBY9JQ2A}{elastic.node2}{192.168.1.24:9300}{m}{xpack.installed=true, transform.node=false}}
[2022-06-29T22:10:46,995][DEBUG][o.e.x.s.a.e.ReservedRealm] [elastic.coordinate.com] realm [reserved] authenticated user [elastic], with roles [[superuser]] (cached)

===========================

Below is the configuration we used to configure the LDAP with Elastic

xpack:
security:
authc:
realms:
active_directory:
my_ad:
order: 0
domain_name: ldapserver
url: ldap://ldapserver:389
files:
role_mapping: /opt/elk/elasticsearch/config/role_mapping.yml

6

There is no indication that the LDAP authentication issue has anything to do with your cluster shutting down.

I suggest we focus on why your cluster shuts down and then if there is a remaining issue with LDAP , we can tackle that too.

Are those the only lines in your log file ? Is this the only node in the cluster ? Your nodes should be logging certain things before they shut down

7

we have 6 nodes in the cluster, and we do see this error in coordinate, master, hot and warm nodes ... the other logs are as below

[2022-06-29T22:10:44,094][DEBUG][o.e.x.s.a.RealmsAuthenticator] [elastic.coordinate.com] Authentication of [elastic] using realm [reserved/reserved] with token [UsernamePasswordToken] was [AuthenticationResult{status=SUCCESS, user=User[username=elastic,roles=[superuser],fullName=null,email=null,metadata={_reserved=true}], message=null, exception=null}]

8

That is not an error. It is an informational message in your logs telling you which realm authenticated the elastic user. That's a sign that things are working correctly.
It is reported at DEBUG level, which means it isn't even included in the logs by default - at some point you have opted to turn on debug logging.

You need to find the last few lines from the log just before the node shuts down. That ought to give some indication of why your cluster is unstable.

If you logs report anything at ERROR or WARN level, that might help diagnose the problem as well.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.