Unable to load org.elasticsearch.xpack.core.XPackPlugin

Elastic Stack Elasticsearch
1

Dear All,
Configuring AD realm and having a problem of unable to load security plugin in 7.3 version. I do not want to load any SSL certificates as of now and neither want to enable SSL. Initially want to test a plain AD authentication and move on step by step. Need your help, it says some '.' is missing

xpack.http.proxy.host: 'my_proxy_IP'
xpack.http.proxy.port: 3128
xpack.watcher.enabled: true
xpack.security.enabled: true
xpack.security.audit.enabled: true
xpack.security.authc.realms.enabled: true
xpack.security.authc.realms.active_directory.example:
 order: 0
 domain_name: example.com
 url: ldap://adserver.example.com:389
 bind_dn: CN=esadmin1,CN=Users,DC=example,DC=com
 bind_password: admin_password
 user_search.base_dn: dc=example,dc=com
 group_search.base_dn: dc=example,dc=com
 files.role_mapping: $ES_PATH_CONF/role_mapping.yml

[root@eshostname elasticsearch]# /usr/share/elasticsearch/bin/elasticsearch-plugin list -v
Plugins directory: /usr/share/elasticsearch/plugins
[root@eshostname elasticsearch]# ll

Error below :

[2019-09-11T23:55:13,718][ERROR][o.e.b.Bootstrap          ] [eshostname] Exception
java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.node.Node.<init>(Node.java:314) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.3.1.jar:7.3.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.3.1.jar:7.3.1]
Caused by: java.lang.reflect.InvocationTargetException
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.3.1.jar:7.3.1]
        ... 15 more
Caused by: org.elasticsearch.common.settings.SettingsException: Failed to get setting group for [xpack.security.authc.realms.] setting prefix and setting [xpack.security.authc.realms.enabled] because of a missing '.'
        at org.elasticsearch.common.settings.Settings.getGroupsInternal(Settings.java:451) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.common.settings.Settings.getGroups(Settings.java:439) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.common.settings.Settings.getGroups(Settings.java:426) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.xpack.core.ssl.SSLService.getRealmsSSLSettings(SSLService.java:646) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:420) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:119) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:146) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
[2019-09-11T23:55:13,718][ERROR][o.e.b.Bootstrap          ] [eshostname] Exception
java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.node.Node.<init>(Node.java:314) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]     at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
2

You stopped the stack trace right before the actual cause is shown, share a few more lines of that error with us.

There is no such setting, please remove this.

bind_password is deprecated and will be removed in a later version. Please use secure_bind_pasword in the secure settings as we describe in our docs

3

due to character sizing limit had to paste only few lines, In this reply I have continued the remaining log contents in next reply

ok. after removing

xpack.security.authc.realms.enabled: true

and modifying

secure_bind_pasword

all previous errors are gone but service is not starting up and has below errors, secondly though it tell about keystore should we first need to create and then add usernames (all?) present in LDAP or which user account need to be added here?

Also wonder like we register (i.e. join the realm) the linux host with AD using admin account, similarly is there any registration mechanism first with which admin account of Elastic Stack services with AD?

[2019-09-12T02:01:06,781][INFO ][o.e.n.Node               ] [eshost] version[7.3.1], pid[20826], build[default/rpm/4749ba6/2019-08-19T20:19
:25.651794Z], OS[Linux/3.10.0-957.27.2.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.2/12.0.2+10]
[2019-09-12T02:01:06,781][INFO ][o.e.n.Node               ] [eshost] JVM home [/usr/share/elasticsearch/jdk]
[2019-09-12T02:01:06,781][INFO ][o.e.n.Node               ] [eshost] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiat
ingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+Always
PreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.n
etty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.t
mpdir=/tmp/elasticsearch-5990270681121175535, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasti
csearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.loca
le.providers=COMPAT, -Dio.netty.allocator.type=pooled, -XX:MaxDirectMemorySize=1073741824, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/e
tc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2019-09-12T02:01:07,991][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [aggs-matrix-stats]
[2019-09-12T02:01:07,992][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [analysis-common]
[2019-09-12T02:01:07,992][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [data-frame]
[2019-09-12T02:01:07,992][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [flattened]
[2019-09-12T02:01:07,992][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [ingest-common]
[2019-09-12T02:01:07,992][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [ingest-geoip]
[2019-09-12T02:01:07,992][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [ingest-user-agent]
[2019-09-12T02:01:07,993][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [lang-expression]
[2019-09-12T02:01:07,993][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [lang-mustache]
[2019-09-12T02:01:07,993][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [lang-painless]
[2019-09-12T02:01:07,993][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [mapper-extras]
[2019-09-12T02:01:07,993][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [parent-join]
[2019-09-12T02:01:07,993][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [percolator]
[2019-09-12T02:01:07,993][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [rank-eval]
[2019-09-12T02:01:07,993][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [reindex]
[2019-09-12T02:01:07,994][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [repository-url]
[2019-09-12T02:01:07,994][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [transport-netty4]
[2019-09-12T02:01:07,994][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [vectors]
[2019-09-12T02:01:07,994][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [x-pack-ccr]
[2019-09-12T02:01:07,994][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [x-pack-core]
[2019-09-12T02:01:07,994][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [x-pack-deprecation]
[2019-09-12T02:01:07,995][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [x-pack-graph]
[2019-09-12T02:01:07,995][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [x-pack-ilm]
[2019-09-12T02:01:07,995][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [x-pack-logstash]
[2019-09-12T02:01:07,995][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [x-pack-ml]
[2019-09-12T02:01:07,995][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [x-pack-monitoring]
[2019-09-12T02:01:07,995][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [x-pack-rollup]
[2019-09-12T02:01:07,995][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [x-pack-security]
[2019-09-12T02:01:07,995][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [x-pack-sql]

Continued in next reply .....

4

... log continued from above response

[2019-09-12T02:01:07,996][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [x-pack-voting-only-node]
[2019-09-12T02:01:07,996][INFO ][o.e.p.PluginsService     ] [eshost] loaded module [x-pack-watcher]
[2019-09-12T02:01:07,996][INFO ][o.e.p.PluginsService     ] [eshost] no plugins loaded
[2019-09-12T02:01:09,711][ERROR][o.e.b.Bootstrap          ] [eshost] Exception
java.lang.IllegalArgumentException: Setting [xpack.security.authc.realms.active_directory.example.secure_bind_password] is a secure setting and must be stored inside the Elasticsearch keystore, but was found inside elasticsearch.yml
        at org.elasticsearch.common.settings.SecureSetting.get(SecureSetting.java:88) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:560) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:476) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:447) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:418) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.node.Node.<init>(Node.java:357) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.3.1.jar:7.3.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.3.1.jar:7.3.1]
[2019-09-12T02:01:09,717][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [eshost] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Setting [xpack.security.authc.realms.active_directory.example.secure_bind_password] is a secure setting and must be stored inside the Elasticsearch keystore, but was found inside elasticsearch.yml
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.3.1.jar:7.3.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.3.1.jar:7.3.1]
Caused by: java.lang.IllegalArgumentException: Setting [xpack.security.authc.realms.active_directory.example.secure_bind_password] is a secure setting and must be stored inside the Elasticsearch keystore, but was found inside elasticsearch.yml
        at org.elasticsearch.common.settings.SecureSetting.get(SecureSetting.java:88) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:560) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:476) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:447) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:418) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.node.Node.<init>(Node.java:357) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.1.jar:7.3.1]
        ... 6 more
5

Please see the docs Ioannis linked to. The secure_bind_password needs to be in the Elasticsearch keystore, not elasticsearch.yml.

6

The error is printed clearly on the logs. secure_bind_password is a secure setting and needs to be added in the keystore.

It's all described in the docs I shared earlier, please read them .

7

ok sure my bad. I have added the admin account password in keystore but this time different error.

[root@eshost elasticsearch]# elasticsearch-keystore list
esadmin1
keystore.seed
xpack.security.authc.realms.active_directory.hdp.secure_bind_password
[root@eshost elasticsearch]#

[2019-09-12T06:27:38,201][ERROR][o.e.b.Bootstrap          ] [eshost] Exception
java.lang.IllegalArgumentException: unknown secure setting [esadmin1] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:531) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:476) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:447) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:418) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.node.Node.<init>(Node.java:357) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.3.1.jar:7.3.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.3.1.jar:7.3.1]
[2019-09-12T06:27:38,207][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [eshost] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown secure setting [esadmin1] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.3.1.jar:7.3.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.3.1.jar:7.3.1]
Caused by: java.lang.IllegalArgumentException: unknown secure setting [esadmin1] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:531) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:476) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:447) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:418) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.node.Node.<init>(Node.java:357) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.1.jar:7.3.1]
        ... 6 more
8

Again the error is self explanatory

This esadmin1 setting you have added is not an allowed secure setting and you need to remove it

9

ok I removed esadmin1 from keystore and now the error disappeared but unable to authenticate AD users accessing elasticsearch with below error. I can login with same userid/password into linux shell prompt but can't login to elasticsearch

[root@eshost elasticsearch]# [2019-09-12T17:02:57,756][WARN ][o.e.x.s.a.AuthenticationService] [eshost] Authentication to realm example failed - authenticate failed (Caused by LDAPException(resultCode=49 (invalid credentials), diagnosticMessage='80090308: LdapErr: DSID-0C09042A, comment: AcceptSecurityContext error, data 52e, v3839', ldapSDKVersion=4.0.8, revision=28812))

is anything wrong with elasticsearch config?

xpack.security.enabled: true
xpack.security.audit.enabled: true
xpack.security.authc.realms.active_directory.example:
 order: 0
 domain_name: example.com
 url: ldap://server1.example.com:389
 bind_dn: CN=Users,DC=example,DC=com
 user_search.base_dn: cn=Users,dc=example,dc=com
 group_search.base_dn: cn=Users,dc=example,dc=com
 files.role_mapping: /etc/elasticsearch/role_mapping.yml


[root@eshost elasticsearch]# elasticsearch-keystore list
keystore.seed
xpack.security.authc.realms.active_directory.example.secure_bind_password
[root@eshost elasticsearch]#

esadmin is the group in AD & esadmin1 & 2 are users of that group in AD

power_user:
#  - "cn=esadmin,dc=example,dc=com"
  - "CN=Users,DC=example,DC=com"
  - "CN=esadmin,CN=Users,DC=example,DC=com"
user:
  - "cn=users,dc=example,dc=com"
#  - "cn=esadmin,dc=example,dc=com"
  - "CN=esadmin,CN=Users,DC=example,DC=com"
  - "cn=esadmin1,cn=esadmin,cn=Users,dc=example,dc=com"
10

How exactly ? An ldapsearch or something else ?

The error is again specific :

resultCode=49 (invalid credentials)

this has nothing to do with role mappings.

11

Since my linux box is integrated with AD, and all those accounts & groups are in AD, I am able to login on linux box with a/c esadmin2 or 1. Same way how we login as root into linux box as below

login as: esadmin2
Keyboard-interactive authentication prompts from server:
| Password:
End of keyboard-interactive prompts from server
Creating home directory for esadmin2.
[esadmin2@eshost ~]$

Agree that's why I checked this method to use same credentials to login into Linux box with same credentials rather than elasticsearch and it was successful and failed in elasticsearch

Is there anything wrong or missing in below configs

xpack.security.enabled: true
xpack.security.audit.enabled: true
xpack.security.authc.realms.active_directory.example:
 order: 0
 domain_name: example.com
 url: ldap://server1.example.com:389
 bind_dn: CN=Users,DC=example,DC=com
 user_search.base_dn: cn=Users,dc=example,dc=com
 group_search.base_dn: cn=Users,dc=example,dc=com
 files.role_mapping: /etc/elasticsearch/role_mapping.yml
13

I got below message when I cancel authentication prompt from a browser, anything to do with charset?

{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [esadmin2] for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"unable to authenticate user [esadmin2] for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}
14

No.

This is definitely wrong. The bind_dn needs to be the DN of a user that has the necessary permissions to search for users and groups in your Active Directory, and the secure_bind_password needs to be the password of that user.

What you have set is not the DN of a user but that of an Organization Unit, which is a different thing. If you don't have a user that you can use as a service account, then you can select to remove the bind_dn and secure_bind_password from the config so all the operations against AD are performed with the authenticating user. Read the doc I have shared above for more details

15

Yes right. after adding the appropriate AD administrator credentials to secure_bind_password the credential error is not there but the problem now is with the role_mapping. I did added below mapping and still I get the below error

monitoring:
  - "CN=esadmin,CN=Users,DC=example,DC=com"
power_user:
  - "CN=Users,DC=example,DC=com"
  - "CN=esadmin,CN=Users,DC=example,DC=com"
user:
  - "cn=users,dc=example,dc=com"
  - "CN=esadmin,CN=Users,DC=example,DC=com"
  - "cn=esadmin1,cn=esadmin,cn=Users,dc=example,dc=com"

I access this URL using "esadmin2" user credentials which is part of esadmin group in AD "http://192.168.1.1:9200/_cluster/health?pretty" below error :

{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "action [cluster:monitor/health] is unauthorized for user [esadmin2]"
      }
    ],
    "type" : "security_exception",
    "reason" : "action [cluster:monitor/health] is unauthorized for user [esadmin2]"
  },
  "status" : 403
}
16

CN=esadmin,CN=Users,DC=example,DC=com doesn't look like it's a group. This is the DN of the esadmin user. So there is no mapping that gives the monitoring role to the esadmin2 user and this is why you get an authorization error.

Also CN=Users,DC=example,DC=com is not a group so you can't use it in role mapping as it is.

Also cn=esadmin1,cn=esadmin,cn=Users,dc=example,dc=com is not a correct DN so you can't use it in role mappings.

The only correct entries in the file you have shared are the ones that give power_user and user to the esadmin user. You need to fix your role mapping.

17

correct as you mentioned those groups were not present. I freshly created few new users & new OU called Groups and within that another OU called esgroup which holds both ELK admins & ELK users. accordingly I modified below mapping. Still its same error
users - elkuser1 & elkuser2
admins - elkadmin1 & elkadmin2

Not sure whats wrong

  1. Appropriate roles not getting mapped which is leading to unauthorized access, I am missing any other configuration like roles.yml?
monitoring:
  - "OU=esgroup,OU=Groups,DC=example,DC=com"
  - "CN=elkadmin1,CN=Users,DC=example,DC=com"
  - "CN=elkadmin2,CN=Users,DC=example,DC=com"
power_user:
  - "CN=elkadmins,OU=esgroup,OU=Groups,DC=example,DC=com"
  - "CN=elkuser1,CN=Users,DC=example,DC=com"
user:
  - "CN=Users,DC=example,DC=com"
  - "CN=elkadmins,OU=esgroup,OU=Groups,DC=example,DC=com"
  - "CN=elkuser1,CN=Users,DC=example,DC=com"

curl -u elkadmin1 "http://192.168.1.1:9200/_cluster/health?pretty"

[2019-09-13T04:21:22,086][DEBUG][o.e.x.s.a.l.ActiveDirectorySessionFactory] [eshost] Resolving LDAP groups + meta-data for user [CN=elkadmin1,CN=Users,DC=example,DC=com]
[2019-09-13T04:21:22,094][DEBUG][o.e.x.s.a.l.ActiveDirectorySessionFactory] [eshost] group SID to DN [CN=elkadmin1,CN=Users,DC=example,DC=com] search filter: [(|(objectSid=S-1-5-32-545)(objectSid=S-1-5-21-3458080126-2923426969-367288162-513)(objectSid=S-1-5-21-3458080126-2923426969-367288162-1128))]
[2019-09-13T04:21:22,096][DEBUG][o.e.x.s.a.l.ActiveDirectorySessionFactory] [eshost] Resolved 1 LDAP groups [[CN=elkadmins,OU=esgroup,OU=Groups,DC=example,DC=com]] for user [CN=elkadmin1,CN=Users,DC=example,DC=com]
[2019-09-13T04:21:22,097][DEBUG][o.e.x.s.a.l.ActiveDirectorySessionFactory] [eshost] Resolved 0 meta-data fields [{}] for user [CN=elkadmin1,CN=Users,DC=example,DC=com]
[2019-09-13T04:21:22,099][DEBUG][o.e.x.s.a.s.DnRoleMapper ] [eshost] the roles [[elastic, power_user, user]], are mapped from these [active_directory] groups [[CN=elkadmins,OU=esgroup,OU=Groups,DC=example,DC=com]] using file [role_mapping.yml] for realm [active_directory/example]
[2019-09-13T04:21:22,099][DEBUG][o.e.x.s.a.s.DnRoleMapper ] [eshost] the roles [[elastic, admin]], are mapped from the user [cn=elkadmin1,cn=users,dc=example,dc=com] using file [role_mapping.yml] for realm [active_directory/example]
[2019-09-13T04:21:22,099][INFO ][o.e.x.s.a.s.m.NativeRoleMappingStore] [eshost] The security index is not yet available - no role mappings can be loaded
[2019-09-13T04:21:22,100][DEBUG][o.e.x.s.a.s.m.NativeRoleMappingStore] [eshost] Security Index [.security] [exists: false] [available: false] [mapping up to date: true]
[2019-09-13T04:21:22,105][DEBUG][o.e.x.s.a.s.m.NativeRoleMappingStore] [eshost] Mapping user [UserData{username:elkadmin1; dn:CN=elkadmin1,CN=Users,DC=example,DC=com; groups:[CN=elkadmins,OU=esgroup,OU=Groups,DC=example,DC=com]; metadata:{ldap_dn=CN=elkadmin1,CN=Users,DC=example,DC=com, ldap_groups=[CN=elkadmins,OU=esgroup,OU=Groups,DC=example,DC=com]}; realm=example}] to roles [[]]
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "action [cluster:monitor/health] is unauthorized for user [elkadmin1]"
      }
    ],
    "type" : "security_exception",
    "reason" : "action [cluster:monitor/health] is unauthorized for user [elkadmin1]"
  },
  "status" : 403
}
  1. Another thing noticed that '.security' index is also not present/available not sure whether it had been created initially when ES was enabled for security as Kibana is not yet been enabled for security. If kibana is down then how would I get this index created
[2019-09-13T04:21:22,099][INFO ][o.e.x.s.a.s.m.NativeRoleMappingStore] [eshost] The security index is not yet available - no role mappings can be loaded
[2019-09-13T04:21:22,100][DEBUG][o.e.x.s.a.s.m.NativeRoleMappingStore] [eshost] Security Index [.security] [exists: false] [available: false] [mapping up to date: true]
  1. Kibana is also not getting started -- log posting in next reply due to character limitation
18

Kibana error from previous post

{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","root"],"pid":10664,"message":"setting up root"}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","server"],"pid":10664,"message":"setting up server"}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","http"],"pid":10664,"message":"starting NotReady server"}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","http","server","Kibana"],"pid":10664,"message":"registering route handler for [/core]"}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","elasticsearch-service"],"pid":10664,"message":"Setting up elasticsearch service"}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","elasticsearch-service"],"pid":10664,"message":"Creating elasticsearch clients"}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","plugins-service"],"pid":10664,"message":"Setting up plugins service"}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","plugins-discovery"],"pid":10664,"message":"Discovering plugins..."}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","plugins-discovery"],"pid":10664,"message":"Scanning \"/usr/share/kibana/src/plugins\" for plugin sub-directories..."}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","plugins-discovery"],"pid":10664,"message":"Scanning \"/usr/share/kibana/x-pack/plugins\" for plugin sub-directories..."}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","plugins-discovery"],"pid":10664,"message":"Scanning \"/usr/share/kibana/plugins\" for plugin sub-directories..."}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","plugins-discovery"],"pid":10664,"message":"Scanning \"/usr/share/kibana-extra\" for plugin sub-directories..."}
{"type":"log","@timestamp":"2019-09-12T18:35:13Z","tags":["debug","plugins-discovery"],"pid":10664,"message":"Successfully discovered plugin \"translations\" at \"/usr/share/kibana/x-pack/plugins/translations\""}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["debug","plugins","translations"],"pid":10664,"message":"\"/usr/share/kibana/x-pack/plugins/translations/server\" does not export \"config\"."}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["debug","plugins-service"],"pid":10664,"message":"Discovered 1 plugins."}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["info","plugins-system"],"pid":10664,"message":"Setting up [1] plugins: [translations]"}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["debug","plugins-system"],"pid":10664,"message":"Setting up plugin \"translations\"..."}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["debug","plugins","translations"],"pid":10664,"message":"Initializing plugin"}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["info","plugins","translations"],"pid":10664,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["debug","root"],"pid":10664,"message":"starting root"}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["debug","plugins-service"],"pid":10664,"message":"Plugins service starts plugins"}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["info","plugins-system"],"pid":10664,"message":"Starting [1] plugins: [translations]"}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["debug","plugins-system"],"pid":10664,"message":"Starting plugin \"translations\"..."}
{"type":"log","@timestamp":"2019-09-12T18:35:14Z","tags":["debug","legacy-service"],"pid":10664,"message":"starting legacy service"}
{"type":"log","@timestamp":"2019-09-12T18:35:17Z","tags":["plugin","debug"],"pid":10664,"path":"/usr/share/kibana/x-pack","message":"Found plugin at /usr/share/kibana/x-pack"}
{"type":"log","@timestamp":"2019-09-12T18:35:17Z","tags":["plugin","debug"],"pid":10664,"path":"/usr/share/kibana/src/legacy/core_plugins/apm_oss","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/apm_oss"}
{"type":"log","@timestamp":"2019-09-12T18:35:17Z","tags":["debug","root"],"pid":10664,"message":"shutting root down"}
19

ok this one got resolved now.

now left with problem 2 & 3. If anyone can help that will be great

20

This is not an error, it's logged on INFO level. You don't have to create the .security index , it will be created automatically when needed. What's the status of your cluster ? What's the output of http://192.168.1.1:9200/_cluster/health?pretty ? Any errors in your logs ?

21 
[root@eshost elasticsearch]# curl -u elkadmin2 "http://192.168.1.1:9200/_cluster/health?pretty"
Enter host password for user 'elkadmin2':
{
  "cluster_name" : "test",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 5,
  "active_shards" : 10,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
[root@eshost elasticsearch]#