Failed to initialize a TrustManager for the system keystore (using elasticsearch-oss; i.e. no XPack)

Alex_Nelson · May 2, 2022, 9:35pm

I manage Elasticsearch on multiple instances.
One of my instances is having problems bringing up Elasticsearch.
I am unable to determine why this one instance is having problems.
All of my instances should have the same versions of the ELK stack.

Note that I'm using the Elasticsearch-oss so no XPack. My company does not allow me to use the basic license.

Also note that I am using a non-bundled version of Java (one that has the recent Java security patches):

JVM home [/usr/lib/jvm/java-17-openjdk-amd64], using bundled JDK [false]

$ java --version
openjdk 17.0.1 2021-10-19
OpenJDK Runtime Environment (build 17.0.1+12-Ubuntu-118.04)
OpenJDK 64-Bit Server VM (build 17.0.1+12-Ubuntu-118.04, mixed mode, sharing)

I am not even using SSL, but still Elasticsearch seeming wants to use SSL (see the stacktrace below).

I am able to access the Elasticsearch keystore using /usr/share/Elasticsearch/bin/Elasticsearch-keystore.

Maybe the "system keystore" is different from the "Elasticsearch keystore" ?

$ whoami
elasticsearch
$ /usr/share/elasticsearch/bin/elasticsearch-keystore has-passwd
ERROR: Keystore is not password-protected

[2022-05-02T16:38:15,181][INFO ][o.e.n.Node               ] [es-efs-master] version[7.10.2], pid[17276], build[oss/deb/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/5.4.0-97-generic/amd64],\
 JVM[Private Build/OpenJDK 64-Bit Server VM/17.0.1/17.0.1+12-Ubuntu-118.04]
[2022-05-02T16:38:15,184][INFO ][o.e.n.Node               ] [es-efs-master] JVM home [/usr/lib/jvm/java-17-openjdk-amd64], using bundled JDK [false]
[2022-05-02T16:38:15,184][INFO ][o.e.n.Node               ] [es-efs-master] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.h\
eadless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacity\
PerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms8025m, -Xmx8025m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:Initiat\
ingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-12235284262591118259, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc\
*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xlog:disable, -XX:MaxDirectMemorySize=4208984064, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elastic\
search, -Des.distribution.flavor=oss, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2022-05-02T16:38:15,889][INFO ][o.e.p.PluginsService     ] [es-efs-master] loaded module [aggs-matrix-stats]
[2022-05-02T16:38:15,889][INFO ][o.e.p.PluginsService     ] [es-efs-master] loaded module [analysis-common]
[2022-05-02T16:38:15,889][INFO ][o.e.p.PluginsService     ] [es-efs-master] loaded module [geo]
[2022-05-02T16:38:15,890][INFO ][o.e.p.PluginsService     ] [es-efs-master] loaded module [ingest-common]
[2022-05-02T16:38:15,890][INFO ][o.e.p.PluginsService     ] [es-efs-master] loaded module [ingest-geoip]
[2022-05-02T16:38:15,890][INFO ][o.e.p.PluginsService     ] [es-efs-master] loaded module [ingest-user-agent]
[2022-05-02T16:38:15,890][INFO ][o.e.p.PluginsService     ] [es-efs-master] loaded module [kibana]
[2022-05-02T16:38:15,890][INFO ][o.e.p.PluginsService     ] [es-efs-master] loaded module [lang-expression]
[2022-05-02T16:38:15,890][INFO ][o.e.p.PluginsService     ] [es-efs-master] loaded module [lang-mustache]
[2022-05-02T16:38:15,891][INFO ][o.e.p.PluginsService     ] [es-efs-master] loaded module [lang-painless]
[2022-05-02T16:38:15,891][INFO ][o.e.p.PluginsService     ] [es-efs-master] loaded module [mapper-extras]
[2022-05-02T16:38:15,891][INFO ][o.e.p.PluginsService     ] [es-efs-master] loaded module [parent-join]
[2022-05-02T16:38:15,891][INFO ][o.e.p.PluginsService     ] [es-efs-master] loaded module [percolator]
[2022-05-02T16:38:15,891][INFO ][o.e.p.PluginsService     ] [es-efs-master] loaded module [rank-eval]
[2022-05-02T16:38:15,892][INFO ][o.e.p.PluginsService     ] [es-efs-master] loaded module [reindex]
[2022-05-02T16:38:15,892][INFO ][o.e.p.PluginsService     ] [es-efs-master] loaded module [repository-url]
[2022-05-02T16:38:15,892][INFO ][o.e.p.PluginsService     ] [es-efs-master] loaded module [systemd]
[2022-05-02T16:38:15,892][INFO ][o.e.p.PluginsService     ] [es-efs-master] loaded module [transport-netty4]
2022-05-02T16:38:15,916][INFO ][o.e.e.NodeEnvironment    ] [es-efs-master] using [1] data paths, mounts [[/home (/dev/md4)]], net usable_space [225.7gb], net total_space [429.7gb], types [ext4]
[2022-05-02T16:38:15,916][INFO ][o.e.e.NodeEnvironment    ] [es-efs-master] heap size [7.8gb], compressed ordinary object pointers [true]
[2022-05-02T16:38:16,171][INFO ][o.e.n.Node               ] [es-efs-master] node name [es-efs-master], node ID [WpZEe_6BQgi4jahGXm4OAQ], cluster name [editshare_monitoring], roles [master, remote_cluster_client, data, inges\
t]
[2022-05-02T16:38:18,159][ERROR][o.e.b.Bootstrap          ] [es-efs-master] Exception
org.elasticsearch.common.ssl.SslConfigException: failed to initialize a TrustManager for the system keystore
        at org.elasticsearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:70) ~[?:?]
        at org.elasticsearch.common.ssl.SslConfiguration.createSslContext(SslConfiguration.java:136) ~[?:?]
        at org.elasticsearch.index.reindex.ReindexSslConfig.reload(ReindexSslConfig.java:145) ~[?:?]
        at org.elasticsearch.index.reindex.ReindexSslConfig.<init>(ReindexSslConfig.java:115) ~[?:?]
        at org.elasticsearch.index.reindex.ReindexPlugin.createComponents(ReindexPlugin.java:91) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$15(Node.java:553) ~[elasticsearch-7.10.2.jar:7.10.2]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:557) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.node.Node.<init>(Node.java:289) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.10.2.jar:7.10.2]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.10.2.jar:7.10.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.10.2.jar:7.10.2]
Caused by: java.security.KeyStoreException: problem accessing trust store
        at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:73) ~[?:?]
        at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:282) ~[?:?]
        at org.elasticsearch.common.ssl.KeyStoreUtil.createTrustManager(KeyStoreUtil.java:151) ~[?:?]
        at org.elasticsearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:68) ~[?:?]
        ... 24 more
Caused by: java.io.IOException: DerInputStream.getLength(): lengthTag=109, too big.
        at sun.security.util.DerInputStream.getLength(DerInputStream.java:251) ~[?:?]

TimV · May 3, 2022, 12:25am

It definitely is.
The system keystore is shipped with your JDK.
In this case, because it's the truststore that's the issue, it's the cacerts file that's being consulted.

It will be in $JAVA_HOME/lib/security/cacerts

Sometimes Linux distributions repackages the cacerts file to match the trusted certificate list that the OS ships (so your JVM, curl, browser, etc all trust the same CAs). That might be the case here since you're not using the ES bundled JDK.

Also, ES 7.10 doesn't officially support JDK17 so it's possible you'll run into other issues there.

Alex_Nelson · May 3, 2022, 1:58am

Excellent! Thank you for the quick response. Indeed the system certs file is different on that one server.

Do you know if there is a way to instruct Elasticsearch to use a different certs file?

Or is it the case that once I've pointed Elasticsearch at a JDK, that's it, I have to use the certs associated with the JDK?

TimV · May 3, 2022, 6:02am

You can add -Djavax.net.ssl.trustStore=/path/to/certs to your jvm options

Alex_Nelson · May 3, 2022, 1:44pm

Thank you again Tim. I'm very grateful for your response. You have made my day.

system · May 31, 2022, 1:45pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.