Attempting to enable TLS

[Russell_Fulton]

Elasticsearch 6.8.12

I have an existing cluster which was set up before x-pack was free. I now want to setup the security "properly". I have used Letsencrypt to install certificates on each of the cluster members and I am following these instructions which are actually for 5.X.

I am currently trying to bring up a single isolated node to do some testing but it dies with the following error:

failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]

But when I start up without the xpack configuration I can see all the xpack modules loaded.

[warkolm (Mark Walkom)]

Can you please post the entire log from node startup.

[Russell_Fulton]

[2020-11-16T11:14:37,837][INFO ][o.e.e.NodeEnvironment    ] [secmonprd08] using [1] data paths, mounts [[/data (data)]], net usable_space [1.1tb], net total_space [7tb], types [zfs]
[2020-11-16T11:14:37,841][INFO ][o.e.e.NodeEnvironment    ] [secmonprd08] heap size [11.4gb], compressed ordinary object pointers [true]
[2020-11-16T11:14:37,843][INFO ][o.e.n.Node               ] [secmonprd08] node name [secmonprd08], node ID [saM1g-kRR22Y0d4N7M0jwQ]
[2020-11-16T11:14:37,843][INFO ][o.e.n.Node               ] [secmonprd08] version[6.8.12], pid[15469], build[default/deb/7a15d2a/2020-08-12T07:27:20.804867Z], OS[Linux/4.15.0-122-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_121/25.121-b13]
[2020-11-16T11:14:37,844][INFO ][o.e.n.Node               ] [secmonprd08] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-4171184815988091138, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb]
[2020-11-16T11:14:39,059][ERROR][o.e.b.Bootstrap          ] [secmonprd08] Exception
java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]

followed by a vast amount of stack trace. There seemed to be some sort of failure in the exception handling.

[warkolm (Mark Walkom)]

Can you please post the entire log.

[Russell_Fulton]

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-6.8.12.jar:6.8.12]

at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.12.jar:6.8.12]

at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.12.jar:6.8.12]

at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.12.jar:6.8.12]

at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.12.jar:6.8.12]

at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.12.jar:6.8.12]

at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.12.jar:6.8.12]

at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.12.jar:6.8.12]

at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.12.jar:6.8.12]

at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.12.jar:6.8.12]

at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.12.jar:6.8.12]

at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.12.jar:6.8.12]

at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.12.jar:6.8.12]

at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.12.jar:6.8.12]

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.12.jar:6.8.12]

at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.12.jar:6.8.12]

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]

at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121]

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.12.jar:6.8.12]

... 15 more

Caused by: org.elasticsearch.ElasticsearchException: failed to create trust manager

at org.elasticsearch.xpack.core.ssl.TrustConfig$CombiningTrustConfig.createTrustManager(TrustConfig.java:126) ~[?:?]

at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]

at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:427) ~[?:?]

at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]

at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]

at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121]

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.12.jar:6.8.12]

... 15 more

Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory

at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.createTrustManager(PEMKeyConfig.java:106) ~[?:?]

at org.elasticsearch.xpack.core.ssl.TrustConfig$CombiningTrustConfig.lambda$createTrustManager$0(TrustConfig.java:122) ~[?:?]

at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267) ~[?:1.8.0_121]

at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_121]

at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_121]

at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_121]

at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_121]

at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_121]

at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_121]

at org.elasticsearch.xpack.core.ssl.TrustConfig$CombiningTrustConfig.createTrustManager(TrustConfig.java:123) ~[?:?]

at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]

at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:427) ~[?:?]

at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]

at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]

at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121]

at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.12.jar:6.8.12]

... 15 more

[Russell_Fulton]

Caused by: java.nio.file.NoSuchFileException: /etc/elasticsearch/ssl/secmonprd08.its.auckland.ac.nz/fullchain1.pem
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
	at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_121]
	at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_121]
	at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:1.8.0_121]
	at java.nio.file.Files.newInputStream(Files.java:152) ~[?:1.8.0_121]
	at org.elasticsearch.xpack.core.ssl.CertParsingUtils.readCertificates(CertParsingUtils.java:93) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.CertParsingUtils.readCertificates(CertParsingUtils.java:86) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.getCertificateChain(PEMKeyConfig.java:71) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.createTrustManager(PEMKeyConfig.java:103) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.TrustConfig$CombiningTrustConfig.lambda$createTrustManager$0(TrustConfig.java:122) ~[?:?]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267) ~[?:1.8.0_121]
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_121]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_121]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_121]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_121]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_121]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_121]
	at org.elasticsearch.xpack.core.ssl.TrustConfig$CombiningTrustConfig.createTrustManager(TrustConfig.java:123) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:427) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.12.jar:6.8.12]
	... 15 more
[2020-11-16T11:14:39,091][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [secmonprd08] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.12.jar:6.8.12]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.12.jar:6.8.12]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.12.jar:6.8.12]
	... 6 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.12.jar:6.8.12]
	... 6 more
Caused by: org.elasticsearch.ElasticsearchException: failed to create trust manager
	at org.elasticsearch.xpack.core.ssl.TrustConfig$CombiningTrustConfig.createTrustManager(TrustConfig.java:126) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:427) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.12.jar:6.8.12]
	... 6 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
	at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.createTrustManager(PEMKeyConfig.java:106) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.TrustConfig$CombiningTrustConfig.lambda$createTrustManager$0(TrustConfig.java:122) ~[?:?]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267) ~[?:1.8.0_121]
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_121]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_121]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_121]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_121]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_121]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_121]
	at org.elasticsearch.xpack.core.ssl.TrustConfig$CombiningTrustConfig.createTrustManager(TrustConfig.java:123) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:427) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.12.jar:6.8.12]
	... 6 more

[Russell_Fulton]

Caused by: java.nio.file.NoSuchFileException: /etc/elasticsearch/ssl/secmonprd08.its.auckland.ac.nz/fullchain1.pem
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
	at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_121]
	at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_121]
	at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:1.8.0_121]
	at java.nio.file.Files.newInputStream(Files.java:152) ~[?:1.8.0_121]
	at org.elasticsearch.xpack.core.ssl.CertParsingUtils.readCertificates(CertParsingUtils.java:93) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.CertParsingUtils.readCertificates(CertParsingUtils.java:86) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.getCertificateChain(PEMKeyConfig.java:71) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.createTrustManager(PEMKeyConfig.java:103) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.TrustConfig$CombiningTrustConfig.lambda$createTrustManager$0(TrustConfig.java:122) ~[?:?]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267) ~[?:1.8.0_121]
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_121]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_121]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_121]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_121]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_121]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_121]
	at org.elasticsearch.xpack.core.ssl.TrustConfig$CombiningTrustConfig.createTrustManager(TrustConfig.java:123) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:394) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:427) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:103) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:144) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_121]
	at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.12.jar:6.8.12]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.12.jar:6.8.12]
	... 6 more

Whew, like a said a lot of stack history!

[warkolm (Mark Walkom)]

@Russell_Fulton

Caused by: java.nio.file.NoSuchFileException: /etc/elasticsearch/ssl/secmonprd08.its.auckland.ac.nz/fullchain1.pem

Did you check that?

[Russell_Fulton]

clearly not : ( I did look through the stack history for clues but I had no idea what I was looking for.

Ah! stack trace is last call at then end -- I searched from the top! Will know for next time!

I mostly use ruby which normally prints them the other way!

I guessed it must be something simple.

Thanks for your help and patience Mark!

[Russell_Fulton]

OK one step further forward and much improved understanding of decoding Java errors.
I now have this working but there was one more wrinkle I will document...

Once I corrected the filename I got permission errors:

Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/elasticsearch/ssl//secmonprd08.its.auckland.ac.nz/fullchain.pem" "read")

ES docs for 6.8 say of the files "This must be a location within the Elasticsearch configuration directory", but it isn't clear to me exactly what that means in the context of unix file system -- in particular does it include symlinks outside.

The answer appears to be no. I tried putting a symlink 'ssl' in /etc/elasticsearch pointing to /etc/letsencrypt/live/secmonprd08 but this still gave the permission error. However removing the symlink and replacing it with a directory and copying the files into it worked.

As you can see this certificate is from Letsencrypt and is managed by certbot so I have inherited the location and structure of the directories. I am reluctant move the files elsewhere since that will interfere with certbot renewing the certificate but it appears I have no choice.

Certbot does have post-deploy script that will handle this sort of thing.