Fail to configure Xpack AD Realms

Hi there,

I'm using ELK 7.6 with xpack security. The error I'm getting is

Authentication to realm foo_ad failed - authenticate failed (Caused by LDAPException(resultCode=49 (invalid credentials), diagnosticMessage='80090308: LdapErr: DSID-0C0903D3, comment: AcceptSecurityContext error, data 52e, v3839', ldapSDKVersion=4.0.8, revision=28812))

I have a Windows AD with this user settings where esuser is the bind user and testuser is use to login to kibana:

I have set elasticsearch.yml like this:

xpack:
  security:
    authc:
      realms:
        native:
          es_native:
            order: 0
        active_directory:
          foo_ad:
            order: 1
            domain_name: ad.foo.local
            url: ldap://ad.foo.local:389
            bind_dn: esuser@foo.local

I have tried removing native realm or change bind_dn to

bind_dn: "cn=esuser,cn=Users,dc=foo,dc=local" 

but same error.

I have also set the bind user password with:

/usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.authc.realms.active_directory.foo_ad.secure_bind_password

I can confirm the settings, bind user password and connectivity with the AD is ok as I've used ldapsearch to connect from ES node:

ldapsearch -x -b "dc=foo,dc=local" -H ldap://ad.foo.local -D "cn=esuser,cn=Users,dc=foo,dc=local" -W

What am I missing here?

Thanks in advance,
CK

You have showed that the esuser credentials work which is greate but you have not mentioned anything regarding how you are attempting to authenticate with testuser and how this is failing.

  • How are you trying to authenticate with testuser ? Directly with curl or in a browser ? via kibana ?
  • Which format are you using for the username ? As in what do you pass in as a username ?

Please set logger.org.elasticsearch.xpack.security.authc.ldap to TRACE and share some of the logs here with us.

Hi There,

The logs for AD realms error is

Authentication to realm foo_ad failed - authenticate failed (Caused by LDAPException(resultCode=49 (invalid credentials), diagnosticMessage='80090308: LdapErr: DSID-0C0903D3, comment: AcceptSecurityContext error, data 52e, v3839^@', ldapSDKVersion=4.0.8, revision=28812))

I believe the error is due to authentication with AD using esuser? I have not progress to using testuser yet. Anyways, testuser is going to be used for login to kibana and I have set a role_mapping file for it.

superuser:
  - "cn=testuser,cn=Users,dc=foo,dc=local"

Thanks
ck

I'm pretty sure there is much more there :slight_smile:

Why do you believe this ?

Then how are you hitting this problem ? How is this triggered ?

Hi,

I hope I got it right. The whole bunch of logs is:

[2020-04-28T15:22:48,416][WARN ][o.e.x.s.a.AuthenticationService] [node-1] Authentication to realm foo_ad failed - authenticate failed (Caused by LDAPException(resultCode=49 (invalid credentials), diagnosticMessage='80090308: LdapErr: DSID-0C0903D3, comment: AcceptSecurityContext error, data 52e, v3839', ldapSDKVersion=4.0.8, revision=28812))
[2020-04-28T15:22:49,237][ERROR][o.e.x.s.a.e.ReservedRealm] [node-1] failed to retrieve password hash for reserved user [kibana]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable
at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:182) ~[x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:525) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:212) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:93) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:167) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:104) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$15(AuthenticationService.java:374) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:102) [x-pack-core-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:419) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$11(AuthenticationService.java:341) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:351) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:292) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:316) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:272) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:254) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:380) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:250) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:310) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:321) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:245) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:122) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:55) [x-pack-security-7.6.2.jar:7.6.2]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:227) [elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:309) [elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:167) [elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:322) [elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:372) [elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:301) [elasticsearch-7.6.2.jar:7.6.2]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:54) [transport-netty4-client-7.6.2.jar:7.6.2]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:29) [transport-netty4-client-7.6.2.jar:7.6.2]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.6.2.jar:7.6.2]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:326) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:300) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) [netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1478) [netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) [netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) [netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:600) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:554) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.43.Final.jar:4.1.43.Final]
at java.lang.Thread.run(Thread.java:830) [?:?]

Doesn't "Authentication to realm foo_ad failed" sounds like the bind user can't connect to the AD? I have not started using testuser to authenticate yet when this is being logged.

The errors are captured when starting ES service.

Thanks
ck

When you start your cluster and your ad realm is higher priority than native or when native is not even defined in your configuration, when you kibana will try to communicate to elatsicsearch, elasticsearch will assume that this is an AD user and try to authenticate kibana against your AD and this would fail. This is what I see in the logs above.

  • What version are your running ?
  • What license do you have installed ?
  • Have you enabled security in elasticsearch.yml ?
  • Have you setup security for your cluster ? i.e. have you set the passwords for the built in user ?
  • What is your current elasticsearch.yml ?

Hi

I'm using ELK 7.6. I have Plat license and have enabled security in elasticsearch.yml.

Yes i have set passwords for all built-in users.

This is a 3-node ES cluster and their elasticsearch.yml files are the same. The relevant portions in elasticsearch.yml looks like this:

xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: certs/es1.key
xpack.security.http.ssl.certificate: certs/es1.crt
xpack.security.http.ssl.certificate_authorities: certs/ca.crt
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.key: certs/es1.key
xpack.security.transport.ssl.certificate: certs/es1.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca.crt
xpack:
  security:
    authc:
      realms:
        native:
          es_native:
            order: 0
        active_directory:
          foo_ad:
            order: 1
            domain_name: ad.foo.local
            url: ldap://ad.foo.local:389
            bind_dn: esuser@foo.local

The security portion looks ok to me. I can login https with elastic user. Logstash and kibana service accounts can also be connected to ES. I can see from discovery dashboard that logs are coming in.

Thanks,
ck

The errors you see that are AD related are a red herring. Kibana attempted to connect to elasticsearch as kibana user ( or you did via kibana login /curl ) and because the security index was not available, Elasticsearch attempted to authenticate the kibana user via your AD realm ( which , as expected, failed. This doesn't indicate that there is something wrong with your AD realm or with the credentials of your esuser. I'm worried we've focused on this specific line of log out of context and we're trying to solve a problem you don't have.

The problem seems to be that native realm is unavailable because your cluster is not healthy.

Now, you say that Kibana can connect to Elasticsearch and you can log in to Kibana and see logs in there, so I can only assume that the error you shared from the logs happened on startup as kibana attemtped to connect to elasticsearch before the security index was available and now this is not the case ( if this is not true, then there would be much more information printed in your logs, so please share them with us ). I'm worried

Can you try and authenticate to elasticsearch as an AD user ? i.e. with

curl -k -utestuser@foo.local https://yourelasticsearchhost:9200/_security/_authenticate

and see what happens ? Alternatively try logging in as testuser@foo.local from kibana

Hi

This is login with testuser.

image

This is the output with the curl command with testuser:

sudo curl -k --cacert /etc/elasticsearch/certs/ca.crt -u testuser@foo.local 'https://10.10.10.41:9200/_security/_authenticate'
Enter host password for user 'testuser@foo.local':
{"error":{"root_cause":[{"type":"security_exception","reason":"error attempting to authenticate request","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}}],"type":"security_exception","reason":"error attempting to authenticate request","caused_by":{"type":"l_d_a_p_exception","reason":"0000208D: NameErr: DSID-03100241, problem 2001 (NO_OBJECT), data 0, best match of:\n\t'DC=foo,DC=local'\n\u0000"},"header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}},"status":401}

This is the log output for testuser:

[2020-04-28T18:49:55,695][WARN ][o.e.x.s.a.AuthenticationService] [node-1] An error occurred while attempting to authenticate [testuser@foo.local] against realm [foo_ad]
com.unboundid.ldap.sdk.LDAPException: 0000208D: NameErr: DSID-03100241, problem 2001 (NO_OBJECT), data 0, best match of:
        'DC=foo,DC=local'

        at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils.toException(LdapUtils.java:407) ~[x-pack-security-7.6.2.jar:7.6.2]
        at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils.access$200(LdapUtils.java:53) ~[x-pack-security-7.6.2.jar:7.6.2]
        at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils$LdapSearchResultListener.searchResultReceived(LdapUtils.java:514) [x-pack-security-7.6.2.jar:7.6.2]
        at com.unboundid.ldap.sdk.AsyncSearchHelper.responseReceived(AsyncSearchHelper.java:245) [unboundid-ldapsdk-4.0.8.jar:4.0.8]
        at com.unboundid.ldap.sdk.LDAPConnectionReader.run(LDAPConnectionReader.java:516) [unboundid-ldapsdk-4.0.8.jar:4.0.8]

Alternately this is the output for curl command with kibana user:

sudo curl -k --cacert /etc/elasticsearch/certs/ca.crt -u kibana:P@ssw0rd 'https://10.10.10.41:9200/_security/_authenticate'
{"username":"kibana","roles":["kibana_system"],"full_name":null,"email":null,"metadata":{"_reserved":true},"enabled":true,"authentication_realm":{"name":"reserved","type":"reserved"},"lookup_realm":{"name":"reserved","type":"reserved"}}

Thanks,
ck

Can you please

I assumed that the testuser's UPN is testuser@foo.local but if it's not, you might want to use testuser as the username ( assuming the sAMAccountName of the testuser is that ) . In general if you share the ldif of your testuser, we might be able to make some more educated guesses on what the configuration needs to be in order to match your AD config . Also, please do read our docs in https://www.elastic.co/guide/en/elasticsearch/reference/current/active-directory-realm.html#ad-realm-configuration, I think you will find them helpful in any case !

I have change the elasticsearch.yml to ldap instead of active_directory and appears to be making some headway.

elasticsearch.yml:

xpack:
  security:
    authc:
      realms:
        native:
          es_native:
            order: 0
#        active_directory:
        ldap:
          foo_ad:
            order: 1
            url: ldap://ad.foo.local:389
            bind_dn: "cn=esuser,cn=Users,dc=foo,dc=local"
            user_search:
              base_dn: "cn=Users,dc=foo,dc=local"
            files:
              role_mapping: "/etc/elasticsearch/role_mapping.yml"

The cluster log displays:

[2020-04-29T21:34:21,318][INFO ][o.e.x.s.a.l.LdapUserSearchSessionFactory] [node-1] Realm [foo_ad] is in user-search mode - base_dn=[cn=Users,dc=foo,dc=local], search filter=[(uid={0})]
[2020-04-29T21:34:21,429][INFO ][o.e.x.s.a.s.FileRolesStore] [node-1] parsed [0] roles from file [/etc/elasticsearch/roles.yml]

However I still cannot login nor make API calls with testuser or testuser@foo.local.

sudo curl -XGET --cacert /etc/elasticsearch/certs/ca.crt -u testuser@foo.local:P@ssw0rd 'https://10.10.10.43:9200/_cluster/health?pretty'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "unable to authenticate user [testuser@foo.local] for REST request [/_cluster/health?pretty]",
        "header" : {
          "WWW-Authenticate" : [
            "Bearer realm=\"security\"",
            "ApiKey",
            "Basic realm=\"security\" charset=\"UTF-8\""
          ]
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "unable to authenticate user [testuser@foo.local] for REST request [/_cluster/health?pretty]",
    "header" : {
      "WWW-Authenticate" : [
        "Bearer realm=\"security\"",
        "ApiKey",
        "Basic realm=\"security\" charset=\"UTF-8\""
      ]
    }
  },
  "status" : 401
}

sudo curl -XGET --cacert /etc/elasticsearch/certs/ca.crt -u testuser:P@ssw0rd 'https://10.10.10.43:9200/_cluster/health?pretty'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "unable to authenticate user [testuser] for REST request [/_cluster/health?pretty]",
        "header" : {
          "WWW-Authenticate" : [
            "Bearer realm=\"security\"",
            "ApiKey",
            "Basic realm=\"security\" charset=\"UTF-8\""
          ]
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "unable to authenticate user [testuser] for REST request [/_cluster/health?pretty]",
    "header" : {
      "WWW-Authenticate" : [
        "Bearer realm=\"security\"",
        "ApiKey",
        "Basic realm=\"security\" charset=\"UTF-8\""
      ]
    }
  },
  "status" : 401
}

When I do a ldapsearch for testuser, the following is given:

# testuser, Users, foo.local
dn: CN=testuser,CN=Users,DC=foo,DC=local
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: testuser
givenName: testuser
distinguishedName: CN=testuser,CN=Users,DC=foo,DC=local
instanceType: 4
whenCreated: 20200424123011.0Z
whenChanged: 20200424123011.0Z
displayName: testuser
uSNCreated: 12796
uSNChanged: 12801
name: testuser
objectGUID:: aYPl+ZaPsEGmbwUFHs/psg==
userAccountControl: 66048
badPwdCount: 0
codePage: 0
countryCode: 0
badPasswordTime: 0
lastLogoff: 0
lastLogon: 0
pwdLastSet: 132322050114257624
primaryGroupID: 513
objectSid:: AQUAAAAAAAUVAAAA6Q5jLhS4kmMCbp8gUAQAAA==
accountExpires: 9223372036854775807
logonCount: 0
sAMAccountName: testuser
sAMAccountType: 805306368
userPrincipalName: testuser@foo.local
objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=foo,DC=local
dSCorePropagationData: 16010101000000.0Z

What worries me is this line in log file "parsed [0] roles from file [/etc/elasticsearch/roles.yml]". Did elasticsearch parse my mappings file at /etc/elasticsearch/role_mapping.yml cos I don't see it in the logs?

Tell me I'm getting somewhere here... :grin:

thanks,
ck

I strongly suggest that you stick with an AD realm instead of an LDAP realm.
I don't see any difference. As I mentioned above, the log line you were seeing on startup with AD realm is not an error, just a red herring.

You still can't authenticate at all so let's focus on this first instead of trying to solve authorization issues that you don't know that you have already :slight_smile:

Can you please revert to an active_directory realm and follow my suggestions above, try to authenticate and share the logs ? Let's try to solve one problem at a time without making unrelated changes. There is a reason why your ldap configuration doesn't work but there is no need to go into details now.

Ok I've switched back to active_directory. Logs are as per usual:

[2020-04-30T00:13:18,200][WARN ][o.e.x.s.a.AuthenticationService] [node-1] Authentication to realm foo_ad failed - authenticate failed (Caused by LDAPException(resultCode=49 (invalid credentials), diagnosticMessage='80090308: LdapErr: DSID-0C0903D3, comment: AcceptSecurityContext error, data 52e, v3839', ldapSDKVersion=4.0.8, revision=28812))
[2020-04-30T00:14:06,906][WARN ][o.e.x.s.a.AuthenticationService] [node-1] Authentication to realm foo_ad failed - authenticate failed (Caused by LDAPException(resultCode=32 (no such object), errorMessage='0000208D: NameErr: DSID-03100241, problem 2001 (NO_OBJECT), data 0, best match of:
        'DC=foo,DC=local'
', matchedDN='DC=foo,DC=local', ldapSDKVersion=4.0.8, revision=28812))
[2020-04-30T00:19:06,662][WARN ][o.e.x.s.a.AuthenticationService] [node-1] Authentication to realm foo_ad failed - authenticate failed (Caused by LDAPException(resultCode=49 (invalid credentials), diagnosticMessage='80090308: LdapErr: DSID-0C0903D3, comment: AcceptSecurityContext error, data 52e, v3839', ldapSDKVersion=4.0.8, revision=28812))
[2020-04-30T00:23:16,689][WARN ][o.e.x.s.a.AuthenticationService] [node-1] Authentication to realm foo_ad failed - authenticate failed (Caused by LDAPException(resultCode=49 (invalid credentials), diagnosticMessage='80090308: LdapErr: DSID-0C0903D3, comment: AcceptSecurityContext error, data 52e, v3839', ldapSDKVersion=4.0.8, revision=28812))

Yes I have set trace:

PUT /_cluster/settings 
{ 
  "transient": { 
     "logger.org.elasticsearch.xpack.security.authc.ldap":"TRACE", 
   } 
}

Here's the ldif file for testuser

dn: CN=testuser,CN=Users,DC=foo,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: testuser
givenName: testuser
distinguishedName: CN=testuser,CN=Users,DC=foo,DC=local
instanceType: 4
whenCreated: 20200424123011.0Z
whenChanged: 20200424123011.0Z
displayName: testuser
uSNCreated: 12796
uSNChanged: 12801
name: testuser
objectGUID:: aYPl+ZaPsEGmbwUFHs/psg==
userAccountControl: 66048
badPwdCount: 0
codePage: 0
countryCode: 0
badPasswordTime: 0
lastLogoff: 0
lastLogon: 0
pwdLastSet: 132322050114257624
primaryGroupID: 513
objectSid:: AQUAAAAAAAUVAAAA6Q5jLhS4kmMCbp8gUAQAAA==
accountExpires: 9223372036854775807
logonCount: 0
sAMAccountName: testuser
sAMAccountType: 805306368
userPrincipalName: testuser@foo.local
objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=foo,DC=local
dSCorePropagationData: 16010101000000.0Z

and esuser...

dn: CN=esuser,CN=Users,DC=foo,DC=local
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
cn: esuser
givenName: esuser
distinguishedName: CN=esuser,CN=Users,DC=foo,DC=local
instanceType: 4
whenCreated: 20200424122854.0Z
whenChanged: 20200429161615.0Z
displayName: esuser
uSNCreated: 12789
uSNChanged: 36910
name: esuser
objectGUID:: obghVugeA0Gqh5iIzggVsg==
userAccountControl: 66048
badPwdCount: 0
codePage: 0
countryCode: 0
badPasswordTime: 132326505685286828
lastLogoff: 0
lastLogon: 132326505754479759
pwdLastSet: 132322049342281188
primaryGroupID: 513
objectSid:: AQUAAAAAAAUVAAAA6Q5jLhS4kmMCbp8gTwQAAA==
accountExpires: 9223372036854775807
logonCount: 0
sAMAccountName: esuser
sAMAccountType: 805306368
userPrincipalName: esuser@foo.local
objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=foo,DC=local
dSCorePropagationData: 16010101000000.0Z
lastLogonTimestamp: 132326505754479759

Thanks
ck

The fact that nothing is printed on DEBUG or TRACE level is problematic as it doesn't allow us to troubleshoot this. Could you

  1. GET /_cluster/settings and verify it is applied correctly
  2. Clear the cache just in case with curl -XPOST 'http://<your_es_instance>:9200/_security/realm/foo_ad/_clear_cache'

The

[2020-04-30T00:14:06,906][WARN ][o.e.x.s.a.AuthenticationService] [node-1] Authentication to realm foo_ad failed - authenticate failed (Caused by LDAPException(resultCode=32 (no such object), errorMessage='0000208D: NameErr: DSID-03100241, problem 2001 (NO_OBJECT), data 0, best match of:
        'DC=foo,DC=local'
', matchedDN='DC=foo,DC=local', ldapSDKVersion=4.0.8, revision=28812))

hinted me to the fact that you are using ad.foo.local as the domain name, while your users reside under CN=Users,DC=foo,DC=local. That, in combination with the fact that you don't set user_search.base_dn explicitly, forces elasticsearch to assume that the baseDN for searching for users is DC=ad,DC=foo,DC=local and it attempts to search for users there, and your AD replies with an error saying that DC=ad,DC=foo,DC=local doesn't exist, as it doesn't.

Can you please set domain_name to foo.local and try this again ?

Hi

I have set the ldap log to trace. Previously I set to transcient and the setting is gone when I restart service. Now I've set it to permanent.

I have also changed my elasticsearch.yml to include a base_dn and change the domain_name

xpack:
  security:
    authc:
      realms:
        native:
          es_native:
            order: 0
        active_directory:
          foo_ad:
            order: 1
            domain_name: foo.local
            url: ldap://ad.foo.local:389
            user_search:
              base_dn: "cn=Users,dc=foo,dc=local"
            bind_dn: esuser@foo.local

Here're the error logs...

[2020-05-06T17:52:19,154][DEBUG][o.e.x.s.a.l.s.LdapUtils  ] [node-1] LDAP bind [SimpleBindRequest(dn='logstash_writer@foo.local')] failed for [LDAPConnectionPool(serverSet=FailoverServerSet(serverSets={SingleServerSet(server=ad.foo.local:389, includesAuthentication=false, includesPostConnectProcessing=false)}), maxConnections=20)] - [LDAPException(resultCode=49 (invalid credentials), diagnosticMessage='80090308: LdapErr: DSID-0C0903D3, comment: AcceptSecurityContext error, data 52e, v3839^@', ldapSDKVersion=4.0.8, revision=28812)]
[2020-05-06T17:52:19,155][DEBUG][o.e.x.s.a.l.LdapRealm    ] [node-1] Exception occurred during authenticate for active_directory/foo_ad
com.unboundid.ldap.sdk.LDAPBindException: 80090308: LdapErr: DSID-0C0903D3, comment: AcceptSecurityContext error, data 52e, v3839^@
        at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:2273) ~[unboundid-ldapsdk-4.0.8.jar:4.0.8]
        at com.unboundid.ldap.sdk.LDAPConnectionPool.bindAndRevertAuthentication(LDAPConnectionPool.java:1619) ~[unboundid-ldapsdk-4.0.8.jar:4.0.8]
        at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils$1.lambda$doRun$0(LdapUtils.java:135) ~[x-pack-security-7.6.2.jar:7.6.2]
        at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
        at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils.privilegedConnect(LdapUtils.java:74) ~[x-pack-security-7.6.2.jar:7.6.2]
        at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils$1.doRun(LdapUtils.java:135) [x-pack-security-7.6.2.jar:7.6.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]
        at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils.maybeForkAndRun(LdapUtils.java:100) [x-pack-security-7.6.2.jar:7.6.2]
        at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils.maybeForkThenBindAndRevert(LdapUtils.java:151) [x-pack-security-7.6.2.jar:7.6.2]
        at org.elasticsearch.xpack.security.authc.ldap.ActiveDirectorySessionFactory$ADAuthenticator.authenticate(ActiveDirectorySessionFactory.java:285) [x-pack-security-7.6.2.jar:7.6.2]
        at org.elasticsearch.xpack.security.authc.ldap.ActiveDirectorySessionFactory.getSessionWithPool(ActiveDirectorySessionFactory.java:116) [x-pack-security-7.6.2.jar:7.6.2]
        at org.elasticsearch.xpack.security.authc.ldap.PoolingSessionFactory.session(PoolingSessionFactory.java:96) [x-pack-security-7.6.2.jar:7.6.2]
        at org.elasticsearch.xpack.security.authc.ldap.LdapRealm.lambda$doAuthenticate$1(LdapRealm.java:131) [x-pack-security-7.6.2.jar:7.6.2]
        at org.elasticsearch.xpack.security.authc.ldap.LdapRealm$CancellableLdapRunnable.doRun(LdapRealm.java:314) [x-pack-security-7.6.2.jar:7.6.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]

and

[2020-05-06T17:52:23,260][TRACE][o.e.x.s.a.l.LdapRealm    ] [node-1] invalidating cache for all users in realm [foo_ad]
[2020-05-06T17:52:23,946][DEBUG][o.e.x.s.a.l.s.LdapUtils  ] [node-1] LDAP bind [SimpleBindRequest(dn='logstash_writer@foo.local')] failed for [LDAPConnectionPool(serverSet=FailoverServerSet(serverSets={SingleServerSet(server=ad.foo.local:389, includesAuthentication=false, includesPostConnectProcessing=false)}), maxConnections=20)] - [LDAPException(resultCode=49 (invalid credentials), diagnosticMessage='80090308: LdapErr: DSID-0C0903D3, comment: AcceptSecurityContext error, data 52e, v3839^@', ldapSDKVersion=4.0.8, revision=28812)]
[2020-05-06T17:52:23,946][DEBUG][o.e.x.s.a.l.LdapRealm    ] [node-1] Exception occurred during authenticate for active_directory/foo_ad
com.unboundid.ldap.sdk.LDAPBindException: 80090308: LdapErr: DSID-0C0903D3, comment: AcceptSecurityContext error, data 52e, v3839^@
        at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:2273) ~[unboundid-ldapsdk-4.0.8.jar:4.0.8]
        at com.unboundid.ldap.sdk.LDAPConnectionPool.bindAndRevertAuthentication(LDAPConnectionPool.java:1619) ~[unboundid-ldapsdk-4.0.8.jar:4.0.8]
        at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils$1.lambda$doRun$0(LdapUtils.java:135) ~[x-pack-security-7.6.2.jar:7.6.2]
        at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
        at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils.privilegedConnect(LdapUtils.java:74) ~[x-pack-security-7.6.2.jar:7.6.2]
        at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils$1.doRun(LdapUtils.java:135) [x-pack-security-7.6.2.jar:7.6.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]
        at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils.maybeForkAndRun(LdapUtils.java:100) [x-pack-security-7.6.2.jar:7.6.2]
        at org.elasticsearch.xpack.security.authc.ldap.support.LdapUtils.maybeForkThenBindAndRevert(LdapUtils.java:151) [x-pack-security-7.6.2.jar:7.6.2]
        at org.elasticsearch.xpack.security.authc.ldap.ActiveDirectorySessionFactory$ADAuthenticator.authenticate(ActiveDirectorySessionFactory.java:285) [x-pack-security-7.6.2.jar:7.6.2]
        at org.elasticsearch.xpack.security.authc.ldap.ActiveDirectorySessionFactory.getSessionWithPool(ActiveDirectorySessionFactory.java:116) [x-pack-security-7.6.2.jar:7.6.2]
        at org.elasticsearch.xpack.security.authc.ldap.PoolingSessionFactory.session(PoolingSessionFactory.java:96) [x-pack-security-7.6.2.jar:7.6.2]
        at org.elasticsearch.xpack.security.authc.ldap.LdapRealm.lambda$doAuthenticate$1(LdapRealm.java:131) [x-pack-security-7.6.2.jar:7.6.2]
        at org.elasticsearch.xpack.security.authc.ldap.LdapRealm$CancellableLdapRunnable.doRun(LdapRealm.java:314) [x-pack-security-7.6.2.jar:7.6.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]
[2020-05-06T17:52:23,947][WARN ][o.e.x.s.a.AuthenticationService] [node-1] Authentication to realm foo_ad failed - authenticate failed (Caused by LDAPException(resultCode=49 (invalid credentials), diagnosticMessage='80090308: LdapErr: DSID-0C0903D3, comment: AcceptSecurityContext error, data 52e, v3839^@', ldapSDKVersion=4.0.8, revision=28812))

Thanks,
ck

Hi Ioannis

It works! I forget to turn on my AD for prev post...

Thanks for all your help! :blush:

ck

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.