How to LDAP Connection?

Elastic Stack Elasticsearch
1

Hi All,

I am trying to integrate EFK to LDAP.

My configurations in ES is. "elaticsearch.yml"

//root@uklvadapp417[DEV][elasticsearch] //# cat elasticsearch.yml
//# ======================== Elasticsearch Configuration =========================
//#
//# NOTE: Elasticsearch comes with reasonable defaults for most settings.
//# Before you set out to tweak and tune the configuration, make sure you
//# understand what are you trying to accomplish and the consequences.
//#
//# The primary way of configuring a node is via this file. This template lists
//# the most important settings you may want to configure for a production cluster.
//#
//# Please consult the documentation for further information on configuration options:
//# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
//#
//# ---------------------------------- Cluster -----------------------------------
//#
//# Use a descriptive name for your cluster:
//#
cluster.name: coltefk
//#
//# ------------------------------------ Node ------------------------------------
//#
//# Use a descriptive name for the node:
//#
node.name: node-3
//#
//# Add custom attributes to the node:
//#
path.repo: /apps/es_backups/
//#
//# ----------------------------------- Paths ------------------------------------
//#
//# Path to directory where to store the data (separate multiple locations by comma):
//#
path.data: /apps/elasticsearch/data
//#
//# Path to log files:
//#
path.logs: /apps/elasticsearch/logs
//#
//# ----------------------------------- Memory -----------------------------------
//#
//# Lock the memory on startup:
//#
//#bootstrap.memory_lock: true
//#
//# Make sure that the heap size is set to about half the memory available
//# on the system and that the owner of the process is allowed to use this
//# limit.
//#
//# Elasticsearch performs poorly when the system is swapping the memory.
//#
//# ---------------------------------- Network -----------------------------------
//#
//# Set the bind address to a specific IP (IPv4 or IPv6):
//#
network.host: 10.198.39.5
//#
//# Set a custom port for HTTP:
//#
//#http.port: 9200
//#
//# For more information, consult the network module documentation.
//#
//# --------------------------------- Discovery ----------------------------------
//#
//# Pass an initial list of hosts to perform discovery when new node is started:
//# The default list of hosts is ["127.0.0.1", "[::1]"]
//#
//discovery.zen.ping.unicast.hosts: ["10.198.39.3","10.198.39.4","10.198.39.5"]
//#
//# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
//#
//discovery.zen.minimum_master_nodes: 2
//#
//# For more information, consult the zen discovery module documentation.
//#
//# ---------------------------------- Gateway -----------------------------------
//#
//# Block initial recovery after a full cluster restart until N nodes are started:
//#
//#gateway.recover_after_nodes: 3
//#
//# For more information, consult the gateway module documentation.
//#
//# ---------------------------------- Various -----------------------------------
//#
//# Require explicit names when deleting indices:
//#
//#action.destructive_requires_name: true
//# BEGIN ANSIBLE MANAGED BLOCK
//action.auto_create_index: true
//xpack:
// security:
// authc:
// realms:
// ldap1:
// type: ldap
// order: 0
// url: "ldap://10.20.235.156:389"
// bind_dn: "cn=mycompany.com,ou=WB,ou=apps,o=mycompany.com"
// bind_password: abc123
// user_search:
// base_dn: "ou=users,o=mycompany.com"
// attribute: cn
// group_search:
// base_dn: "ou=users,o=mycompany.com"
// files:
// role_mapping: "/etc/elasticsearch/x-pack/role_mapping.yml"
// unmapped_groups_as_roles: false
//# END ANSIBLE MANAGED BLOCK

My roles_mapping.yml is

//root@uklvadapp417[DEV][x-pack] # cat role_mapping.yml
//# Role mapping configuration file which has elasticsearch roles as keys
//# that map to one or more user or group distinguished names

//#roleA: this is an elasticsearch role
//# - groupA-DN this is a group distinguished name
//# - groupB-DN
//# - user1-DN this is the full user distinguished name

//power_user:
//cn=mycompany.com,ou=GBIL,ou=apps,o=mycompany.com
//#user:
//# - "cn=users,dc=example,dc=com"
//cn=mycompany.com,ou=GBIL,ou=apps,o=mycompany.com
//# - "cn=John Doe,cn=other users,dc=example,dc=com"
//root@uklvadapp417[DEV][x-pack] #

I dont see if I am connected and I cant autheticate with an ldap user.

I have check also:

//coltapps@uklvadapp092[DEV][gb-efk-plays] $ curl -u elastic -XGET -u admin 'http://10.198.39.5:9200/_xpack/usage?pretty'
//Enter host password for user 'elastic':
//Enter host password for user 'admin':
//{
// "security" : {
// "available" : true,
// "enabled" : true,
// "realms" : {
// "file" : {
// "name" : [
// "default_file"
// ],
// "available" : true,
// "size" : [
// 1
// ],
// "enabled" : true,
// "order" : [
// 2147483647
// ]
// },
// "ldap" : {
// "available" : true,
// "enabled" : false
// },
// "native" : {
// "name" : [
// "default_native"
// ],
// "available" : true,
// "size" : [
// 0
// ],
// "enabled" : true,
// "order" : [
// 2147483647
// ]
// },
// "active_directory" : {
// "available" : true,
// "enabled" : false
// },
// "pki" : {
// "available" : true,
// "enabled" : false
// }
// },
// "roles" : {
// "native" : {
// "size" : 0,
// "fls" : false,
// "dls" : false
// },
// "file" : {
// "size" : 0,
// "fls" : false,
// "dls" : false
// }
// },
// "ssl" : {
// "http" : {
// "enabled" : false
// },
// "transport" : {
// "enabled" : false
// }
// },
// "audit" : {
// "outputs" : [
// "logfile"
// ],
// "enabled" : false
// },
// "ipfilter" : {
// "http" : false,
// "transport" : false
// },
// "system_key" : {
// "enabled" : false
// },
// "anonymous" : {
// "enabled" : false
// }
// },
// "watcher" : {
// "available" : true,
// "enabled" : true,
// "count" : {
// "active" : 0,
// "total" : 0
// },
// "execution" : {
// "actions" : {
// "_all" : {
// "total" : 0,
// "total_time_in_ms" : 0
// }
// }
// }
// },
// "monitoring" : {
// "available" : true,
// "enabled" : true,
// "enabled_exporters" : {
// "local" : 1
// }
// },
// "graph" : {
// "available" : true,
// "enabled" : true
// }
}
Please advise.

2

Please use the </> button to reformat your message, it's really really hard to follow your configuration in its current format.

You have to share some more information. How do you attempt to authenticate ? Does the user you try to authenticate with already exist under ou=users,o=mycompany.com in your ldap ?

Try the following:

  • Set ldap logging to debug with

    curl -H "Content-Type: application/json" -XPUT -uelastic 'http://localhost:9200/_cluster/settings' -d'
{
 "transient" : {
     "logger.org.elasticsearch.xpack.security.authc.ldap" : "DEBUG"
 }
}'

  • Attempt to login with an ldap user

  • Share the relevant part of elasticsearch.log ( look in /apps/elasticsearch/logs )

3

coltapps@uklvadapp092[DEV][gb-efk-plays] $ curl -H "Content-Type: application/json" -XPUT -uelastic 'http://10.198.39.8:9200/_cluster/settings' -d'

{
"transient" : {
"logger.org.elasticsearch.xpack.security.authc.ldap" : "DEBUG"
}
}'
Enter host password for user 'elastic':
{"acknowledged":true,"persistent":{},"transient":{"logger":{"org":{"elasticsearch":{"xpack":{"security":{"authc":{"ldap":"DEBUG"}}}}}}}}coltapps@uklvadapp092[DEV][gb-efk-plays] $

4

When i try to connect using my ldap user error is below.

coltapps@uklvadapp092[DEV][gb-efk-plays] $ curl -H "Content-Type: application/json" -XPUT -u SGZ1coltefkusers 'http://10.198.39.8:9200/_cluster/settings' -d'
{
 "transient" : {
     "logger.org.elasticsearch.xpack.security.authc.ldap" : "DEBUG"
 }
}'
Enter host password for user 'SGZ1coltefkusers':
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [SGZ1coltefkusers] for REST request [/_cluster/settings]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"unable to authenticate user [SGZ1coltefkusers] for REST request [/_cluster/settings]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}coltapps@uklvadapp092[DEV][gb-efk-plays] $
5

You need to configure the logging with a user that can authenticate successfully, such as elastic.

6

I do have tried with elastic and it able to authenticate:

coltapps@uklvadapp092[DEV][tasks] $ curl -H "Content-Type: application/json" -XPUT -u elastic 'http://10.198.39.5:9200/_cluster/settings' -d'
> {
>  "transient" : {
>      "logger.org.elasticsearch.xpack.security.authc.ldap" : "TRACE"
>  }
> }'
Enter host password for user 'elastic':
{"acknowledged":true,"persistent":{},"transient":{"logger":{"org":{"elasticsearch":{"xpack":{"security":{"authc":{"ldap":"TRACE"}}}}}}}}coltapps@uklvadapp092[DEV][tasks] $
7

Here is the log when I use the LDAP user:

[2018-03-30T03:13:37,037][DEBUG][r.suppressed             ] path: /_cluster/settings, params: {}
org.elasticsearch.ElasticsearchSecurityException: unable to authenticate user [SGZ1coltefkusers] for REST request [/_cluster/settings]
        at org.elasticsearch.xpack.security.support.Exceptions.authenticationError(Exceptions.java:39) ~[?:?]
        at org.elasticsearch.xpack.security.authc.DefaultAuthenticationFailureHandler.failedAuthentication(DefaultAuthenticationFailureHandler.java:37) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$AuditableRestRequest.authenticationFailed(AuthenticationService.java:537) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeUser(AuthenticationService.java:331) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:56) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.xpack.common.IteratingActionListener.onResponse(IteratingActionListener.java:66) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$null$8(AuthenticationService.java:261) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:56) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$0(CachingUsernamePasswordRealm.java:110) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:56) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$doAuthenticateAndCache$3(CachingUsernamePasswordRealm.java:141) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:56) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.lambda$verifyPassword$8(NativeUsersStore.java:590) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:56) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore$1.onResponse(NativeUsersStore.java:192) ~[?:?]
        at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore$1.onResponse(NativeUsersStore.java:189) ~[?:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:91) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:87) ~[elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:121) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:56) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:247) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:233) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1017) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.transport.TcpTransport$1.doRun(TcpTransport.java:1386) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:109) [elasticsearch-5.2.2.jar:5.2.2]
        at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:1378)
8

cont:

[elasticsearch-5.2.2.jar:5.2.2]
            at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1347) [elasticsearch-5.2.2.jar:5.2.2]
            at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) [transport-netty4-5.2.2.jar:5.2.2]
            at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
            at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
            at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
            at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
            at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
            at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
            at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
            at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
            at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
            at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
            at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
            at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
            at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
            at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
            at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
            at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
            at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:527) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
            at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:481) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
            at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
            at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.7.Final.jar:4.1.7.Final]
            at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
    [2018-03-30T03:13:37,698][DEBUG][o.e.x.s.a.e.ReservedRealm] [node-
9

Hi,

The following is just enabling the DEBUG log for ldap by changing the cluster settings, it was not meant to be an authentication test.

Have you restarted Elasticsearch after defining the ldap realm in elasticsearch.yml ?

Try authenticating as follows:

 curl -u SGZ1coltefkusers 'http://10.198.39.8:9200/_xpack/security/_authenticate?pretty'

and check the logs again.

10

Elasticsearch 5.2.2 is quite old and there have been a lot of improvements in the year or so since it was released. If you are able to run a newer version, that would make things easier for you.

11

Getting a newer version is not in the pipeline right now. Im sure there is better explanation for this.

12

Hi @Christopher_Bogs_Oli,

Tim's suggestion is valid irrespective and not directly related to your current issue. As you can see we have been trying to assist you to get this resolved with your current installation. I still believe this is a configuration issue so please perform the steps that we have outlined for you in above and get back to us with the logs so that we can assist you further

13

Hi,

coltapps@uklvadapp092[DEV][gb-efk-plays] $ curl -H "Content-Type: application/json" -XPUT -uelastic 'http://10.198.39.3:9200/_cluster/settings' -d'

{
"transient" : {
"logger.org.elasticsearch.xpack.security.authc.ldap" : "DEBUG"
}
}'
Enter host password for user 'elastic':
{"acknowledged":true,"persistent":{},"transient":{"logger":{"org":{"elasticsearch":{"xpack":{"security":{"authc":{"ldap":"DEBUG"}}}}}}}}coltapps@uklvadapp092[DEV][gb-efk-plays] $

{"acknowledged":true,"persistent":{},"transient":{"logger":{"org":{"elasticsearch":{"xpack":{"security":{"authc":{"ldap":"DEBUG"}}}}}}}}coltapps@uklvadapp092[DEV][gbcoltapps@uklvadapp092[DEV][gb-efk-plays] $ curl -H "Content-Type: application/json" -XPUT -u SGZ1coltefkusers 'http://10.198.39.5:9200/_cluster/settings' -d'

{
"transient" : {
"logger.org.elasticsearch.xpack.security.authc.ldap" : "DEBUG"
}
}'
Enter host password for user 'SGZ1coltefkusers':
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [SGZ1coltefkusers] for REST request [/_cluster/settings]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"unable to authenticate user [SGZ1coltefkusers] for REST request [/_cluster/settings]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}coltapps@uklvadapp092[DEV][gb-efk-plays] $

14

Hi,

Please read my answer once again and perform the steps that I have asked you instead of re-posting one of your earlier answers. We need you to run the query asked ( against the authentication API ) and share the elasticsearch logs so that we can help you solve your issues.
Finally, please so use the </> button correctly, it is really hard to go through logs and json structures when they are posted as plain text.

15

curl -u SGZ1coltefkusers 'http://10.198.39.5:9200/_xpack/security/_authenticate?pretty'
Enter host password for user 'SGZ1coltefkusers':
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "unable to authenticate user [SGZ1coltefkusers] for REST request [/_xpack/security/_authenticate?pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
}
],
"type" : "security_exception",
"reason" : "unable to authenticate user [SGZ1coltefkusers] for REST request [/_xpack/security/_authenticate?pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="security" charset="UTF-8""
}
},
"status" : 401
}

curl -u elastic 'http://10.198.39.5:9200/_xpack/security/_authenticate?pretty'
Enter host password for user 'elastic':
{
"username" : "elastic",
"roles" : [
"superuser"
],
"full_name" : null,
"email" : null,
"metadata" : {
"_reserved" : true
},
"enabled" : true
}

16

Hi again,

The crucial part that you didn't include is the elasticsearch log, as this is what will tell us what goes wrong. If you don't share this, we can only make wild guesses as to what the issue might be. So, once more, just so that I am clear

  1. Enable debug logging for ldap running the following as the elastic user :

    curl -H "Content-Type: application/json" -XPUT -uelastic 'http://10.198.39.5:9200/_cluster/settings' -d'
{
   "transient" : {
       "logger.org.elasticsearch.xpack.security.authc.ldap" : "DEBUG"
     }
}'

  2. Try to authenticate as one of your ldap users ( i.e. SGZ1coltefkusers )

    curl -u SGZ1coltefkusers 'http://10.198.39.8:9200/_xpack/security/_authenticate?pretty'

  3. Share the relevant part of elasticsearch.log with us so we can see what the issue might be.

17

The short answer is that your realm is not correctly configured. However, we can't offer much more advice than that without your logs.
When Elasticsearch attempts to authenticate SGZ1coltefkusers against your LDAP directory it fails.
That could be caused by a number of issues including:

  • url: "ldap://10.20.235.156:389"
    This URL might be wrong, and the node cannot connect to your directory
  • bind_dn: "cn=mycompany.com,ou=WB,ou=apps,o=mycompany.com"
    bind_password: abc123
    These credentials might be wrong, and the node cannot connect to your directory
  • bind_dn: "cn=mycompany.com,ou=WB,ou=apps,o=mycompany.com"
    The credentials might be correct, but perhaps this user does not have permission to search your directory.
  • user_search.base_dn: "ou=users,o=mycompany.com"
    This DN might be wrong and the SGZ1coltefkusers user doesn't exist under that part of the LDAP tree.
  • attribute: cn
    This attribute might be wrong, and we cannot find a SGZ1coltefkusers user when we search by cn.
  • Or SGZ1coltefkusers might simply not exist at all.
  • Or you might be entering an incorrect password for SGZ1coltefkusers

The set of possibilities is large enough that we cannot solve this for you unless you can provide us with details from your log files.

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.