Elasticsearch 7.1.x fails to start with security enabled

Elastic Stack Elasticsearch
1

Trying to follow the instructions from

After editing the config/elasticsearch.yaml

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

I get the following when I run bin/elasticsearch

[2019-05-30T09:28:16,827][INFO ][o.e.e.NodeEnvironment    ] [bos-mpwo9] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [104.2gb], net total_space [233.5gb], types [apfs]
[2019-05-30T09:28:16,830][INFO ][o.e.e.NodeEnvironment    ] [bos-mpwo9] heap size [989.8mb], compressed ordinary object pointers [true]
[2019-05-30T09:28:16,834][INFO ][o.e.n.Node               ] [bos-mpwo9] node name [bos-mpwo9], node ID [9K-cYbnYTVKGEhJLo183nA], cluster name [elasticsearch]
[2019-05-30T09:28:16,834][INFO ][o.e.n.Node               ] [bos-mpwo9] version[7.1.1], pid[15328], build[default/tar/7a013de/2019-05-23T14:04:00.380842Z], OS[Mac OS X/10.14.5/x86_64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/1.8.0_212/25.212-b03]
[2019-05-30T09:28:16,835][INFO ][o.e.n.Node               ] [bos-mpwo9] JVM home [/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre]
[2019-05-30T09:28:16,835][INFO ][o.e.n.Node               ] [bos-mpwo9] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/var/folders/q_/frmj6zk95y7_vlspkcg9d14r0000gy/T/elasticsearch-2931096571122859559, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Dio.netty.allocator.type=unpooled, -Des.path.home=/Users/kwlee/Downloads/elasticsearch-7.1.1, -Des.path.conf=/Users/kwlee/Downloads/elasticsearch-7.1.1/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2019-05-30T09:28:17,473][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [bos-mpwo9] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.1.1.jar:7.1.1]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.1.1.jar:7.1.1]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.node.Node.<init>(Node.java:308) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.1.jar:7.1.1]
	... 6 more

It seems it is looking for the xpack security plugin. Then I tried to install the xpack

bin/elasticsearch-plugin install x-pack
ERROR: this distribution of Elasticsearch contains X-Pack by default

I think I am missing something.

2

How did you install Elasticsearch?

3

I simply downloaded and extracted the tar.gz file. After modified the elasticsearch.yml, I started it by running bin/elasticsearch. Just as described on https://www.elastic.co/downloads/elasticsearch.

4

There should be additional detail in the log. You've cut off the bit that tells us what went wrong.

5

Hi Tim,

Thanks for getting back to me. Here is the details from the elasticsearch.log.

 [2019-05-30T09:28:16,827][INFO ][o.e.e.NodeEnvironment    ] [bos-mpwo9] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [104.2gb], net total_space [233.5gb], types [apfs]
[2019-05-30T09:28:16,830][INFO ][o.e.e.NodeEnvironment    ] [bos-mpwo9] heap size [989.8mb], compressed ordinary object pointers [true]
[2019-05-30T09:28:16,834][INFO ][o.e.n.Node               ] [bos-mpwo9] node name [bos-mpwo9], node ID [9K-cYbnYTVKGEhJLo183nA], cluster name [elasticsearch]
[2019-05-30T09:28:16,834][INFO ][o.e.n.Node               ] [bos-mpwo9] version[7.1.1], pid[15328], build[default/tar/7a013de/2019-05-23T14:04:00.380842Z], OS[Mac OS X/10.14.5/x86_64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/1.8.0_212/25.212-b03]
[2019-05-30T09:28:16,835][INFO ][o.e.n.Node               ] [bos-mpwo9] JVM home [/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre]
[2019-05-30T09:28:16,835][INFO ][o.e.n.Node               ] [bos-mpwo9] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/var/folders/q_/frmj6zk95y7_vlspkcg9d14r0000gy/T/elasticsearch-2931096571122859559, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Dio.netty.allocator.type=unpooled, -Des.path.home=/Users/kwlee/Downloads/elasticsearch-7.1.1, -Des.path.conf=/Users/kwlee/Downloads/elasticsearch-7.1.1/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2019-05-30T09:28:17,464][ERROR][o.e.b.Bootstrap          ] [bos-mpwo9] Exception
java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.node.Node.<init>(Node.java:308) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) [elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.1.1.jar:7.1.1]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.1.1.jar:7.1.1]
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.1.1.jar:7.1.1]
	... 15 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
	at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:61) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:382) ~[?:?]
	at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_212]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:434) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:119) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.1.1.jar:7.1.1]
	... 15 more
6 
Caused by: java.io.IOException: keystore password was incorrect
	at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2059) ~[?:?]
	at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_212]
	at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:89) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:58) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:382) ~[?:?]
	at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_212]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:434) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:119) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.1.1.jar:7.1.1]
	... 15 more
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
	at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2059) ~[?:?]
	at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_212]
	at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:89) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:58) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:382) ~[?:?]
	at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_212]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:434) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:119) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.1.1.jar:7.1.1]
	... 15 more
[2019-05-30T09:28:17,473][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [bos-mpwo9] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.1.1.jar:7.1.1]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.1.1.jar:7.1.1]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.node.Node.<init>(Node.java:308) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.1.jar:7.1.1]
	... 6 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.node.Node.<init>(Node.java:308) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.1.jar:7.1.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.1.jar:7.1.1]
	... 6 more
7

That's your problem. It appears that you put a password on your PKCS#12 keystore (elastic-certificates.p12) but haven't configured Elasticsearch to use one.

You should add xpack.security.transport.ssl.keystore.password and xpack.security.transport.ssl.truststore.password to your elasticsearch secure settings

8

I did not exactly remember if I had had a password when I generated the certificate. I just regenerate the certificate without password, and it seems working now. Thanks for your help.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.