Kibana & Elastic failing due to LDAP Issue

Hi,

Some of our Kibana instances are shutting down due to an LDAP error that is occuring in ES nodes. The error we receive in Kibana is Request Timeout after 3000ms. The following is the error we are receiving in the kibana logs:

{"type":"log","@timestamp":"2018-07-21T07:55:25Z","tags":["status","plugin:logstash@6.2.3","error"],"pid":76720,"state":"red","message":"Status changed from red to red - [security_exception] unable to authenticate user [*******] for REST request [/_xpack], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } }","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2018-07-21T07:55:25Z","tags":["warning","monitoring-ui","kibana-monitoring"],"pid":76720,"message":"Unable to fetch data from kibana_settings collector"}
{"type":"error","@timestamp":"2018-07-21T07:55:25Z","tags":["warning","monitoring-ui","kibana-monitoring"],"pid":76720,"level":"error","error":{"message":"[security_exception] unable to authenticate user [*******] for REST request [/.kibana/_search?ignore_unavailable=true&filter_path=aggregations.types.buckets], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } }","name":"Error","stack":"[security_exception] unable to authenticate user [*******] for REST request [/.kibana/_search?ignore_unavailable=true&filter_path=aggregations.types.buckets], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/.kibana/_search\",\"query\":{\"ignore_unavailable\":true,\"filter_path\":\"aggregations.types.buckets\"},\"body\":\"{\\\"size\\\":0,\\\"query\\\":{\\\"terms\\\":{\\\"type\\\":[\\\"dashboard\\\",\\\"visualization\\\",\\\"search\\\",\\\"index-pattern\\\",\\\"graph-workspace\\\",\\\"timelion-sheet\\\"]}},\\\"aggs\\\":{\\\"types\\\":{\\\"terms\\\":{\\\"field\\\":\\\"type\\\",\\\"size\\\":6}}}}\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"unable to authenticate user [*******] for REST request [/.kibana/_search?ignore_unavailable=true&filter_path=aggregations.types.buckets]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"unable to authenticate user [*******] for REST request [/.kibana/_search?ignore_unavailable=true&filter_path=aggregations.types.buckets]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}\n    at respond (/opt/appl/kibana/node_modules/elasticsearch/src/lib/transport.js:295:15)\n    at checkRespForFailure (/opt/appl/kibana/node_modules/elasticsearch/src/lib/transport.js:254:7)\n    at HttpConnector.<anonymous> (/opt/appl/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n    at IncomingMessage.bound (/opt/appl/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n    at emitNone (events.js:91:20)\n    at IncomingMessage.emit (events.js:185:7)\n    at endReadableNT (_stream_readable.js:974:12)\n    at _combinedTickCallback (internal/process/next_tick.js:80:11)\n    at process._tickDomainCallback (internal/process/next_tick.js:128:9)"},"message":"[security_exception] unable to authenticate user [*******] for REST request [/.kibana/_search?ignore_unavailable=true&filter_path=aggregations.types.buckets], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } }"}

And in our Elasticsearch logs, we receive the following error:

[WARN ][o.e.x.s.a.l.s.LdapUtils ] caught exception while trying to follow referral [ldaps:///DC=,DC=,DC=]
com.unboundid.ldap.sdk.LDAPException: An error occurred while attempting to send the LDAP message to server : SocketTimeoutException(message='Read timed out', trace='socketRead0(SocketInputStream.java:native) / socketRead(SocketInputStream.java:116) / read(SocketInputStream.java:171) / read(SocketInputStream.java:141) / readFully(InputRecord.java:465) / read(InputRecord.java:503) / readRecord(SSLSocketImpl.java:983) / performInitialHandshake(SSLSocketImpl.java:1385) / writeRecord(SSLSocketImpl.java:757) / write(AppOutputStream.java:123) / flushBuffer(BufferedOutputStream.java:82) / flush(BufferedOutputStream.java:140) / sendMessage(LDAPConnectionInternals.java:543) / sendMessage(LDAPConnection.java:4249) / process(SimpleBindRequest.java:551) / bind(LDAPConnection.java:2143) / getReferralConnection(LDAPConnection.java:4573) / lambda$followReferral$11(LdapUtils.java:601) / doPrivileged(AccessController.java:native) / privilegedConnect(LdapUtils.java:87) / followReferral(LdapUtils.java:601) / access$300(LdapUtils.java:66) / searchResultReceived(LdapUtils.java:533) / responseReceived(AsyncSearchHelper.java:240) / run(LDAPConnectionReader.java:569)', revision=24201)
at com.unboundid.ldap.sdk.LDAPConnectionInternals.sendMessage(LDAPConnectionInternals.java:574) ~[unboundid-ldapsdk-3.2.0.jar:3.2.0]
at com.unboundid.ldap.sdk.LDAPConnection.sendMessage(LDAPConnection.java:4249) ~[unboundid-ldapsdk-3.2.0.jar:3.2.0]
at com.unboundid.ldap.sdk.SimpleBindRequest.process(SimpleBindRequest.java:551) ~[unboundid-ldapsdk-3.2.0.jar:3.2.0]
at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:2143) ~[unboundid-ldapsdk-3.2.0.jar:3.2.0]
at com.unboundid.ldap.sdk.LDAPConnection.getReferralConnection(LDAPConnection.java:4573) ~[unboundid-ldapsdk-3.2.0.jar:3.2.0]
at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils.lambda$followReferral$11(LdapUtils.java:601) ~[x-pack-security-6.2.3.jar:6.2.3]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils.privilegedConnect(LdapUtils.java:87) ~[x-pack-security-6.2.3.jar:6.2.3]
at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils.followReferral(LdapUtils.java:601) ~[x-pack-security-6.2.3.jar:6.2.3]
at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils.access$300(LdapUtils.java:66) ~[x-pack-security-6.2.3.jar:6.2.3]
at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils$LdapSearchResultListener.searchResultReceived(LdapUtils.java:533) [x-pack-security-6.2.3.jar:6.2.3]
at com.unboundid.ldap.sdk.AsyncSearchHelper.responseReceived(AsyncSearchHelper.java:240) [unboundid-ldapsdk-3.2.0.jar:3.2.0]
at com.unboundid.ldap.sdk.LDAPConnectionReader.run(LDAPConnectionReader.java:569) [unboundid-ldapsdk-3.2.0.jar:3.2.0]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_161]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_161]
at java.net.SocketInputStream.read(SocketInputStream.java:171) ~[?:1.8.0_161]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_161]
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) ~[?:?]
at sun.security.ssl.InputRecord.read(InputRecord.java:503) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) ~[?:?]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) ~[?:?]
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:757) ~[?:?]
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) ~[?:?]
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_161]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_161]
at com.unboundid.ldap.sdk.LDAPConnectionInternals.sendMessage(LDAPConnectionInternals.java:543) ~[?:?]
... 12 more

[WARN ][o.e.x.s.a.AuthenticationService] [hostname] Authentication to realm active_directory2 failed - authenticate failed (Caused by LDAPException(resultCode=49 (invalid credentials), errorMessage='80090308: LdapErr: DSID-0C090400, comment: AcceptSecurityContext error, data 52e, v1db1 ', diagnosticMessage='80090308: LdapErr: DSID-0C090400, comment: AcceptSecurityContext error, data 52e, v1db1 '))

[WARN ][o.e.x.s.a.AuthenticationService] [hostname] Authentication to realm active_directory failed - authenticate failed (Caused by LDAPException(resultCode=81 (server down), errorMessage='An error occurred while attempting to send the LDAP message to server : SocketTimeoutException(message='Read timed out', trace='socketRead0(SocketInputStream.java:native) / socketRead(SocketInputStream.java:116) / read(SocketInputStream.java:171) / read(SocketInputStream.java:141) / readFully(InputRecord.java:465) / read(InputRecord.java:503) / readRecord(SSLSocketImpl.java:983) / performInitialHandshake(SSLSocketImpl.java:1385) / writeRecord(SSLSocketImpl.java:757) / write(AppOutputStream.java:123) / flushBuffer(BufferedOutputStream.java:82) / flush(BufferedOutputStream.java:140) / sendMessage(LDAPConnectionInternals.java:543) / sendMessage(LDAPConnection.java:4249) / process(SimpleBindRequest.java:551) / bind(LDAPConnection.java:2143) / doRun(LdapUtils.java:138) / run(AbstractRunnable.java:37) / maybeForkThenBind(LdapUtils.java:161) / authenticate(ActiveDirectorySessionFactory.java:258) / getSessionWithoutPool(ActiveDirectorySessionFactory.java:133) / session(PoolingSessionFactory.java:86) / lambda$doAuthenticate$1(LdapRealm.java:137) / doRun(LdapRealm.java:293) / doRun(ThreadContext.java:672) / run(AbstractRunnable.java:37) / runWorker(ThreadPoolExecutor.java:1149) / run(ThreadPoolExecutor.java:624) / run(Thread.java:748)', revision=24201)'))

[WARN ][o.e.x.s.a.AuthenticationService] [hostname] Authentication to realm active_directory2 failed - authenticate failed (Caused by LDAPException(resultCode=49 (invalid credentials), errorMessage='80090308: LdapErr: DSID-0C090400, comment: AcceptSecurityContext error, data 52e, v1db1 ', diagnosticMessage='80090308: LdapErr: DSID-0C090400, comment: AcceptSecurityContext error, data 52e, v1db1 '))

As far as I could understand, the above LDAP exception occurs due to invalid credentials but there is also the Error 81 (server down) that occurs so could this also occur due to some issue on LDAP side? We have tested our credentials initially and it has been working so far without any issues. This issue only occurs in some servers and it causes quite the problem as Kibana tends to go down due to the request timeout.
Is there any other possible reason this issue could be occurring?

What do you mean exactly by "Kibana goes down". Is the Kibana server process terminating, or is it just an error for the user logging in?
This sort of problem should not cause Kibana to actually shutdown.

This seems to be the core of your problem.
Your main AD server is sending a referral (which is a bit like a redirect) to another AD server that is not reachable.
It is likely to be a misconfiguration in your AD forest, but you could try to workaround it by adding

follow_referrals: false

to your realm configuration.
But make sure you test that first. Depending on the setup of your AD forest, you may need referals turned on in order for authentication and/or group resolution to work correctly.

The server process terminates. The error logs I have posted above occur in the same time frame. Kibana status goes from green to red:

{"type":"log","@timestamp":"2018-07-21T08:36:48Z","tags":["info","monitoring-ui","kibana-monitoring"],"pid":38070,"message":"Stopping all Kibana monitoring collectors"}
{"type":"log","@timestamp":"2018-07-21T08:36:48Z","tags":["status","plugin:elasticsearch@6.2.3","error"],"pid":38070,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
{"type":"error","@timestamp":"2018-07-21T08:36:56Z","tags":["warning","monitoring-ui","kibana-monitoring"],"pid":38070,"level":"error","error":{"message":"[security_exception] unable to authenticate user for REST request [/.kibana/_search?ignore_unavailable=true&filter_path=aggregations.types.buckets], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } }","name":"Error","stack":"[security_exception] unable to authenticate user for REST request [/.kibana/_search?ignore_unavailable=true&filter_path=aggregations.types.buckets], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } } :: {"path":"/.kibana/_search","query":{"ignore_unavailable":true,"filter_path":"aggregations.types.buckets"},"body":"{\"size\":0,\"query\":{\"terms\":{\"type\":[\"dashboard\",\"visualization\",\"search\",\"index-pattern\",\"graph-workspace\",\"timelion-sheet\"]}},\"aggs\":{\"types\":{\"terms\":{\"field\":\"type\",\"size\":6}}}}","statusCode":401,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate user for REST request [/.kibana/_search?ignore_unavailable=true&filter_path=aggregations.types.buckets]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate user [TA1056Q] for REST request [/.kibana/_search?ignore_unavailable=true&filter_path=aggregations.types.buckets]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401}","wwwAuthenticateDirective":"Basic realm=\"security\" charset=\"UTF-8\""}\n at respond (/opt/appl/kibana/node_modules/elasticsearch/src/lib/transport.js:295:15)\n at checkRespForFailure (/opt/appl/kibana/node_modules/elasticsearch/src/lib/transport.js:254:7)\n at HttpConnector. (/opt/appl/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n at IncomingMessage.bound (/opt/appl/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n at emitNone (events.js:91:20)\n at IncomingMessage.emit (events.js:185:7)\n at endReadableNT (_stream_readable.js:974:12)\n at _combinedTickCallback (internal/process/next_tick.js:80:11)\n at process._tickDomainCallback (internal/process/next_tick.js:128:9)"},"message":"[security_exception] unable to authenticate user for REST request [/.kibana/_search?ignore_unavailable=true&filter_path=aggregations.types.buckets], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } }"}

This is because we have provided the LDAP user creds for the ES username and password in kibana.yml.

I tested the LDAP authentication with the TRACE logging on and your deduction of this issue is correct. The logs I got in Elasticsearch are as follows:

[TRACE][o.e.x.s.a.l.s.LdapUtils ] LDAP referred elsewhere SearchRequest(baseDN='DC=,DC=', scope=, deref=NEVER, sizeLimit=0, timeLimit=5, filter='(&(objectClass=user)(|(sAMAccountName=)(userPrincipalName=@.)))', attrs={1.1}) => [ldaps:///DC=,DC=,DC=, ldaps:///DC=,DC=,DC=, ldaps:///CN=Configuration,DC=,DC=]

[WARN ][o.e.x.s.a.l.s.LdapUtils ] caught exception while trying to follow referral [ldaps:///DC=,DC=,DC=]
com.unboundid.ldap.sdk.LDAPException: An error occurred while attempting to connect to server ForestDnsZones. . :636: java.io.IOException: LDAPException(resultCode=91 (connect error), errorMessage='An error occurred while attempting to establish a connection to server : ConnectException(message='Connection refused (Connection refused)
Caused by: java.io.IOException: LDAPException(resultCode=91 (connect error), errorMessage='An error occurred while attempting to establish a connection to server : ConnectException(message='Connection refused (Connection refused)'

Caused by: com.unboundid.ldap.sdk.LDAPException: An error occurred while attempting to establish a connection to server : ConnectException(message='Connection refused (Connection refused)

[TRACE][o.e.x.s.a.l.s.LdapUtils ] LDAP referred elsewhere SearchRequest(baseDN='DC=,DC=', scope=SUB, deref=NEVER, sizeLimit=0, timeLimit=5, filter='(&(objectClass=user)(|(sAMAccountName=)(userPrincipalName=@ . )))', attrs={1.1}) => [ldaps:// . /CN=Schema,CN=Configuration,DC= ,DC= ]

[TRACE][o.e.x.s.a.l.s.LdapUtils ] LDAP Search SearchRequest(baseDN='DC= ,DC= ', scope=SUB, deref=NEVER, sizeLimit=0, timeLimit=5, filter='(&(objectClass=user)(|(sAMAccountName=)(userPrincipalName=@ . )))', attrs={1.1}) => SearchResult(resultCode=0 (success), messageID=2, entriesReturned=0, referencesReturned=0) ()

[TRACE][o.e.x.s.a.l.s.LdapUtils ] LDAP Search SearchRequest(baseDN='CN=,OU=,OU=,OU=,DC= ,DC= ', scope=BASE, deref=NEVER, sizeLimit=0, timeLimit=5, filter='(objectClass=*)', attrs={tokenGroups}) => SearchResult(resultCode=0 (success), messageID=3, entriesReturned=1, referencesReturned=0) ([SearchResultEntry(dn='CN=,OU=,OU=,OU=,DC= ,DC= ', messageID=3, attributes={Attribute(name=tokenGroups, base64Values={'AQIAAAAAAAUgAAAAIQIAAA==',......})}, controls={})])
[2018-07-30T06:13:31,513][TRACE][o.e.x.s.a.l.s.LdapUtils ] LDAP referred elsewhere SearchRequest(baseDN='DC= ,DC= ', scope=SUB, deref=NEVER, sizeLimit=0, timeLimit=5, filter='(|(objectSid=......)', attrs={1.1}) => [ldaps://. . /DC=,DC= ,DC= , ldaps://. . /DC=,DC= ,DC= , ldaps:// . /CN=Configuration,DC= ,DC= ]

[WARN ][o.e.x.s.a.l.s.LdapUtils ] caught exception while trying to follow referral [ldaps://. . /DC=,DC= ,DC= ]
com.unboundid.ldap.sdk.LDAPException: An error occurred while attempting to connect to server . . :: java.io.IOException: LDAPException(resultCode=91 (connect error), errorMessage='An error occurred while attempting to establish a connection to server . . /

[TRACE][o.e.x.s.a.l.s.LdapUtils ] LDAP Search SearchRequest(baseDN='DC= ,DC= ', scope=SUB, deref=NEVER, sizeLimit=0, timeLimit=5, filter='(|(objectSid=......)', attrs={1.1}) => SearchResult(resultCode=0 (success), messageID=2, entriesReturned=0, referencesReturned=0) ()

[TRACE][o.e.x.s.a.l.s.LdapUtils ] LDAP referred elsewhere SearchRequest(baseDN='DC= ,DC= ', scope=SUB, deref=NEVER, sizeLimit=0, timeLimit=5, filter='(|(objectSid=......)', attrs={1.1}) => [ldaps:// . /CN=Schema,CN=Configuration,DC= ,DC= ]
[2018-07-30T06:13:31,541][TRACE][o.e.x.s.a.l.s.LdapUtils ] LDAP Search SearchRequest(baseDN='DC= ,DC= ', scope=SUB, deref=NEVER, sizeLimit=0, timeLimit=5, filter='(|(objectSid=......)', attrs={1.1}) => SearchResult(resultCode=0 (success), messageID=2, entriesReturned=0, referencesReturned=0) ()

Is the only workaround for this to turn off the referrals? What are the other solutions that we could try to fix this issue because it is occurring in multiple nodes.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.