Configuring LDAP Auth causes status=1/FAILURE


(Jose Tobar) #1

I've been going through the documentation for setting up authentication with LDAP but I must be missing something because after adding my changes to the file /etc/elasticsearch/elasticsearch.yml the service refuses to start and if I go to kibana I'm met with:

Login is currently disabled. Administrators should consult the Kibana logs for more details.

This is what my /etc/elasticsearch/elasticsearch.yml file contains:

cluster.name: elasticsearch
node.name: default-amazon
path.data: "/var/lib/elasticsearch"
path.logs: "/var/log/elasticsearch"
xpack.security.authc.realms:
  ldap1:
    type: ldap
    order: 0
    url: "ldaps://openldap.mysweetdomain.com:636"
    bind_dn: "cn=admin,dc=mysweetdomain,dc=com"
    user_search:
      base_dn: "dc=mysweetdomain,dc=com"
    group_search:
      base_dn: "dc=mysweetdomain,dc=com"
    files:
      role_mapping: "CONFIG_DIR/role_mapping.yml"
    unmapped_groups_as_roles: false
    ssl.verification_mode: none

I have a valid certificate but for the sake of testing, I've asked this configuration not to validate it. If I remove the xpack configuration all together it starts up fine and kibana will display but then I have no authentication/authorization in place.

Right now starting up the service results in the following

[root@ip-10-0-0-40 elasticsearch]# service elasticsearch start
Starting elasticsearch (via systemctl):                    [  OK  ]
[root@ip-10-0-0-40 elasticsearch]# service elasticsearch status
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-06-27 19:04:46 UTC; 1s ago
     Docs: http://www.elastic.co
 Main PID: 16118 (java)
   CGroup: /system.slice/elasticsearch.service
           ‣ 16118 [java]

Jun 27 19:04:47 ip-10-0-0-40.us-east-2.compute.internal elasticsearch[16118]: at org.elasticsearch.bootstrap.Bootstrap.loadSecureSettings(Bootstrap.java:226)
Jun 27 19:04:47 ip-10-0-0-40.us-east-2.compute.internal elasticsearch[16118]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:291)
Jun 27 19:04:47 ip-10-0-0-40.us-east-2.compute.internal elasticsearch[16118]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136)
Jun 27 19:04:47 ip-10-0-0-40.us-east-2.compute.internal elasticsearch[16118]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127)
Jun 27 19:04:47 ip-10-0-0-40.us-east-2.compute.internal elasticsearch[16118]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Jun 27 19:04:47 ip-10-0-0-40.us-east-2.compute.internal elasticsearch[16118]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
Jun 27 19:04:47 ip-10-0-0-40.us-east-2.compute.internal elasticsearch[16118]: at org.elasticsearch.cli.Command.main(Command.java:90)
Jun 27 19:04:47 ip-10-0-0-40.us-east-2.compute.internal elasticsearch[16118]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
Jun 27 19:04:47 ip-10-0-0-40.us-east-2.compute.internal elasticsearch[16118]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86)
Jun 27 19:04:47 ip-10-0-0-40.us-east-2.compute.internal elasticsearch[16118]: Refer to the log for complete error details.
[root@ip-10-0-0-40 elasticsearch]# service elasticsearch status
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2018-06-27 19:04:47 UTC; 7s ago
     Docs: http://www.elastic.co
  Process: 16118 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 16118 (code=exited, status=1/FAILURE)

Jun 27 19:04:47 ip-10-0-0-40.us-east-2.compute.internal elasticsearch[16118]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127)
Jun 27 19:04:47 ip-10-0-0-40.us-east-2.compute.internal elasticsearch[16118]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Jun 27 19:04:47 ip-10-0-0-40.us-east-2.compute.internal elasticsearch[16118]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
Jun 27 19:04:47 ip-10-0-0-40.us-east-2.compute.internal elasticsearch[16118]: at org.elasticsearch.cli.Command.main(Command.java:90)
Jun 27 19:04:47 ip-10-0-0-40.us-east-2.compute.internal elasticsearch[16118]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
Jun 27 19:04:47 ip-10-0-0-40.us-east-2.compute.internal elasticsearch[16118]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86)
Jun 27 19:04:47 ip-10-0-0-40.us-east-2.compute.internal elasticsearch[16118]: Refer to the log for complete error details.
Jun 27 19:04:47 ip-10-0-0-40.us-east-2.compute.internal systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jun 27 19:04:47 ip-10-0-0-40.us-east-2.compute.internal systemd[1]: Unit elasticsearch.service entered failed state.
Jun 27 19:04:47 ip-10-0-0-40.us-east-2.compute.internal systemd[1]: elasticsearch.service failed.

It only stays alive for a split second for dying. Any help will be appreciated, if need be I can post an example of my OpenLDAP.

LDAP SERVER STATS:
OpenLDAP 2.4.42 running on Ubuntu x64 16.04

ELK Server:
Amazon Linux LTS 2 x64
ELK is version 6.3.0


(Tim Vernum) #2

What do the elasticsearch logs say?
If elasticsearch is failing to start then the logs will tell you why.


(Jose Tobar) #3

It doesn’t log actually, the Kibana logs say it’s can’t revive the connection. I can post them here.


(Jose Tobar) #4

After I remade the server, I added my changes but it complained about missing roles. so I went ahead and defined those as well, I'll post that so as to make sure with people who actually know about elasticsearch.

[root@ip-10-0-0-11 ec2-user]# cat /etc/elasticsearch/role_mapping.yml
# Role mapping configuration file which has elasticsearch roles as keys
# that map to one or more user or group distinguished names

#roleA:   this is an elasticsearch role
#  - groupA-DN  this is a group distinguished name
#  - groupB-DN
#  - user1-DN   this is the full user distinguished name

ldap1:
 - uid=jxt100,ou=People,dc=mysweetdomain,dc=com

#power_user:
#  - "cn=admins,dc=example,dc=com"
#user:
#  - "cn=users,dc=example,dc=com"
#  - "cn=admins,dc=example,dc=com"
#  - "cn=John Doe,cn=other users,dc=example,dc=com"

It went on to not complain about roles anymore but then it won't start anymore either:

[root@ip-10-0-0-11 ec2-user]# tail -10 /var/log/elasticsearch/elasticsearch
tail: cannot open ‘/var/log/elasticsearch/elasticsearch’ for reading: No such file or directory
[root@ip-10-0-0-11 ec2-user]# tail -10 /var/log/elasticsearch/elasticsearch.log
[2018-06-28T01:57:39,296][INFO ][o.e.n.Node               ] [default-amazon] starting ...
[2018-06-28T01:57:39,443][INFO ][o.e.t.TransportService   ] [default-amazon] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-06-28T01:57:39,466][ERROR][o.e.b.Bootstrap          ] [default-amazon] node validation exception
[1] bootstrap checks failed
[1]: Role mapping file [/etc/elasticsearch/CONFIG_DIR/role_mapping.yml] for realm [ldap1] does not exist.
[2018-06-28T01:57:39,476][INFO ][o.e.n.Node               ] [default-amazon] stopping ...
[2018-06-28T01:57:39,489][INFO ][o.e.n.Node               ] [default-amazon] stopped
[2018-06-28T01:57:39,490][INFO ][o.e.n.Node               ] [default-amazon] closing ...
[2018-06-28T01:57:39,498][INFO ][o.e.n.Node               ] [default-amazon] closed
[2018-06-28T01:57:39,500][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
[root@ip-10-0-0-11 ec2-user]# tail -20 /var/log/elasticsearch/elasticsearch.log
[2018-06-28T01:57:33,754][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [x-pack-upgrade]
[2018-06-28T01:57:33,754][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [x-pack-watcher]
[2018-06-28T01:57:33,755][INFO ][o.e.p.PluginsService     ] [default-amazon] no plugins loaded
[2018-06-28T01:57:37,162][INFO ][o.e.x.s.a.l.LdapUserSearchSessionFactory] [default-amazon] Realm [ldap1] is in user-search mode - base_dn=[dc=mysweetdomain,dc=com], search filter=[(uid={0})]
[2018-06-28T01:57:37,164][WARN ][o.e.x.s.a.s.DnRoleMapper ] [default-amazon] Role mapping file [/etc/elasticsearch/CONFIG_DIR/role_mapping.yml] for realm [ldap1] does not exist. Role mapping will be skipped.
[2018-06-28T01:57:37,179][INFO ][o.e.x.s.a.s.FileRolesStore] [default-amazon] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2018-06-28T01:57:37,981][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/7091] [Main.cc@109] controller (64 bit): Version 6.3.0 (Build 0f0a34c67965d7) Copyright (c) 2018 Elasticsearch BV
[2018-06-28T01:57:38,352][DEBUG][o.e.a.ActionModule       ] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2018-06-28T01:57:38,554][INFO ][o.e.d.DiscoveryModule    ] [default-amazon] using discovery type [zen]
[2018-06-28T01:57:39,296][INFO ][o.e.n.Node               ] [default-amazon] initialized
[2018-06-28T01:57:39,296][INFO ][o.e.n.Node               ] [default-amazon] starting ...
[2018-06-28T01:57:39,443][INFO ][o.e.t.TransportService   ] [default-amazon] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-06-28T01:57:39,466][ERROR][o.e.b.Bootstrap          ] [default-amazon] node validation exception
[1] bootstrap checks failed
[1]: Role mapping file [/etc/elasticsearch/CONFIG_DIR/role_mapping.yml] for realm [ldap1] does not exist.
[2018-06-28T01:57:39,476][INFO ][o.e.n.Node               ] [default-amazon] stopping ...
[2018-06-28T01:57:39,489][INFO ][o.e.n.Node               ] [default-amazon] stopped
[2018-06-28T01:57:39,490][INFO ][o.e.n.Node               ] [default-amazon] closing ...
[2018-06-28T01:57:39,498][INFO ][o.e.n.Node               ] [default-amazon] closed
[2018-06-28T01:57:39,500][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started

I don't believe it should be a memory issue as it has about 4gbs of ram.


(Jose Tobar) #5

UPDATE: To anyone who may also be following this thread, I found that the previous issue about native controller process being stopped and the inability to spawn a new one was due to memory - or not being allocated enough.

Launching the elasticsearch with the following commands alleviates it:

sudo ES_JAVA_OPTS="-Xms2g -Xmx2g" systemctl stop elasticsearch.service

However, after this Kibana and Elasticsearch seem to work just fine but I'm not being prompted to login with my ldap credentials.


(Jose Tobar) #6
[2018-06-28T02:12:20,659][INFO ][o.e.n.Node               ] [default-amazon] initializing ...
[2018-06-28T02:12:20,711][INFO ][o.e.e.NodeEnvironment    ] [default-amazon] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [4.8gb], net total_space [7.9gb], types [xfs]
[2018-06-28T02:12:20,711][INFO ][o.e.e.NodeEnvironment    ] [default-amazon] heap size [1.9gb], compressed ordinary object pointers [true]
[2018-06-28T02:12:20,712][INFO ][o.e.n.Node               ] [default-amazon] node name [default-amazon], node ID [HJkvPUiQRNuPNZUY0Oqb5g]
[2018-06-28T02:12:20,712][INFO ][o.e.n.Node               ] [default-amazon] version[6.3.0], pid[4156], build[default/rpm/424e937/2018-06-11T23:38:03.357887Z], OS[Linux/4.14.33-59.37.amzn2.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_171/25.171-b10]
[2018-06-28T02:12:20,712][INFO ][o.e.n.Node               ] [default-amazon] JVM arguments [-Xms1973m, -Xmx1973m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm]
[2018-06-28T02:12:22,988][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [aggs-matrix-stats]
[2018-06-28T02:12:22,988][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [analysis-common]
[2018-06-28T02:12:22,988][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [ingest-common]
[2018-06-28T02:12:22,989][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [lang-expression]
[2018-06-28T02:12:22,989][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [lang-mustache]
[2018-06-28T02:12:22,989][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [lang-painless]
[2018-06-28T02:12:22,989][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [mapper-extras]
[2018-06-28T02:12:22,989][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [parent-join]
[2018-06-28T02:12:22,989][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [percolator]
[2018-06-28T02:12:22,989][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [rank-eval]
[2018-06-28T02:12:22,989][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [reindex]
[2018-06-28T02:12:22,989][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [repository-url]
[2018-06-28T02:12:22,989][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [transport-netty4]
[2018-06-28T02:12:22,989][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [tribe]
[2018-06-28T02:12:22,989][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [x-pack-core]
[2018-06-28T02:12:22,989][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [x-pack-deprecation]
[2018-06-28T02:12:22,990][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [x-pack-graph]
[2018-06-28T02:12:22,990][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [x-pack-logstash]
[2018-06-28T02:12:22,990][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [x-pack-ml]
[2018-06-28T02:12:22,990][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [x-pack-monitoring]
[2018-06-28T02:12:22,990][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [x-pack-rollup]
[2018-06-28T02:12:22,990][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [x-pack-security]
[2018-06-28T02:12:22,990][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [x-pack-sql]
[2018-06-28T02:12:22,990][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [x-pack-upgrade]
[2018-06-28T02:12:22,990][INFO ][o.e.p.PluginsService     ] [default-amazon] loaded module [x-pack-watcher]
[2018-06-28T02:12:22,991][INFO ][o.e.p.PluginsService     ] [default-amazon] no plugins loaded
[2018-06-28T02:12:26,599][INFO ][o.e.x.s.a.l.LdapUserSearchSessionFactory] [default-amazon] Realm [ldap1] is in user-search mode - base_dn=[dc=mysweetdomain,dc=com], search filter=[(uid={0})]
[2018-06-28T02:12:26,627][INFO ][o.e.x.s.a.s.FileRolesStore] [default-amazon] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2018-06-28T02:12:27,433][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/4209] [Main.cc@109] controller (64 bit): Version 6.3.0 (Build 0f0a34c67965d7) Copyright (c) 2018 Elasticsearch BV
[2018-06-28T02:12:27,820][DEBUG][o.e.a.ActionModule       ] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2018-06-28T02:12:28,064][INFO ][o.e.d.DiscoveryModule    ] [default-amazon] using discovery type [zen]
[2018-06-28T02:12:28,919][INFO ][o.e.n.Node               ] [default-amazon] initialized
[2018-06-28T02:12:28,919][INFO ][o.e.n.Node               ] [default-amazon] starting ...
[2018-06-28T02:12:29,067][INFO ][o.e.t.TransportService   ] [default-amazon] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-06-28T02:12:32,149][INFO ][o.e.c.s.MasterService    ] [default-amazon] zen-disco-elected-as-master ([0] nodes joined)[, ], reason: new_master {default-amazon}{HJkvPUiQRNuPNZUY0Oqb5g}{V6oFjdD4RbWsxa2ZzDZ7ZA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=4138708992, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2018-06-28T02:12:32,154][INFO ][o.e.c.s.ClusterApplierService] [default-amazon] new_master {default-amazon}{HJkvPUiQRNuPNZUY0Oqb5g}{V6oFjdD4RbWsxa2ZzDZ7ZA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=4138708992, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {default-amazon}{HJkvPUiQRNuPNZUY0Oqb5g}{V6oFjdD4RbWsxa2ZzDZ7ZA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=4138708992, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)[, ]]])
[2018-06-28T02:12:32,248][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [default-amazon] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-06-28T02:12:32,248][INFO ][o.e.n.Node               ] [default-amazon] started
[2018-06-28T02:12:32,506][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [default-amazon] Failed to clear cache for realms [[ldap1]]
[2018-06-28T02:12:32,526][INFO ][o.e.l.LicenseService     ] [default-amazon] license [561c2cc6-4e53-4280-b84c-7265edce06fa] mode [basic] - valid
[2018-06-28T02:12:32,536][INFO ][o.e.g.GatewayService     ] [default-amazon] recovered [0] indices into cluster_state

(Jose Tobar) #7

The logs do not seem to indicate if something went wrong with the query :confused:


(Tim Vernum) #8

This is the issue prevent you from logging in.
Your cluster is running with a "basic" license.
The "basic" license does not include security.

See: https://www.elastic.co/subscriptions

If you have a commercial license, then you can apply it to your cluster or you can start a free 30 day trial.
See: https://www.elastic.co/guide/en/kibana/6.3/managing-licenses.html


(Jose Tobar) #9

Ooooh. I misunderstood, thought that since xpack was built into it as of 6.3.x that it was free. Can you guys look into making that error better? Lol but thank you for all your help I’ll either switch the license or I’ll switch to another elastic provider.


(system) #10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.