Getting "signed fields invalid" when configure HTTPS to Elastic HTTP layer with X-Pack

Victor_Guimaraes · July 5, 2019, 7:54pm

Hello,

I'm testing the Elasticsearch 7.2, thinking in upgrade my production cluster from 6.4 version to 7.2.

On my 6.4 cluster, I'm using SearchGuard for security, and works very well. I want to use X-Pack in the new version, but I can't configure HTTPS on Elastic's HTTP layer. I'm using my organization's certificate in a PEM format, the same I'm using on production's SearchGuard and Kibana currently. But on X-Pack I ever get the follow error: CertificateParsingException: signed fields invalid.

My configuration on SearchGuard:

searchguard.ssl.http.pemkey_filepath: certificate.key
searchguard.ssl.http.pemcert_filepath: certificate.pem
searchguard.ssl.http.pemtrustedcas_filepath: certificate.pem

My configuration on X-Pack:

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: certificate.key
xpack.security.http.ssl.certificate: certificate.pem
xpack.security.http.ssl.certificate_authorities: ["certificate.pem"]

I'm certain that the problem is not in the certificate, so I believe I'm configuring the X-Pack HTTPS incorrecly. Someone can help me?

ikakavas · July 5, 2019, 8:12pm

Please share the entire log message and not only part of it . Where do you get this ?

I can only guess that the file you have named as certificate.pem is not actually a PEM encoded X509 certificate, but something else ( maybe a DER encoded certificate or a keystore of some sort?)

Can you share the output of

openssl x509 -in certificate.pem -text -inform der

openssl x509 -in certificate.pem -text -inform pem

Does any of the two above return some output instead of an error ?

Victor_Guimaraes · July 5, 2019, 8:49pm

Sorry not share the log before. I have the limit of 7000 characters and I can't upload files except images, so I must remove log parts.

Message log

[2019-07-05T20:40:48,632][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-staging-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.2.0.jar:7.2.0]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.2.0.jar:7.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.2.0.jar:7.2.0]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.node.Node.<init>(Node.java:307) ~[elasticsearch-7.2.0.jar:7.2.0]
... 6 more
Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
... 6 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
at org.elasticsearch.xpack.core.ssl.PEMTrustConfig.createTrustManager(PEMTrustConfig.java:48) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:382) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1133) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$2(SSLService.java:426) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1333) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:423) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:119) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:143) ~[?:?]
... 6 more
Caused by: java.security.cert.CertificateParsingException: signed fields invalid
at sun.security.x509.X509CertImpl.parse(X509CertImpl.java:1830) ~[?:?]
at sun.security.x509.X509CertImpl.<init>(X509CertImpl.java:188) ~[?:?]
at sun.security.provider.X509Factory.parseX509orPKCS7Cert(X509Factory.java:476) ~[?:?]
at sun.security.provider.X509Factory.engineGenerateCertificates(X509Factory.java:361) ~[?:?]
at java.security.cert.CertificateFactory.generateCertificates(CertificateFactory.java:478) ~[?:?]
at org.elasticsearch.xpack.core.ssl.CertParsingUtils.readCertificates(CertParsingUtils.java:94) ~[?:?]
at org.elasticsearch.xpack.core.ssl.CertParsingUtils.readCertificates(CertParsingUtils.java:86) ~[?:?]
at org.elasticsearch.xpack.core.ssl.PEMTrustConfig.createTrustManager(PEMTrustConfig.java:45) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:382) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1133) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$2(SSLService.java:426) ~[?:?]
at java.util.HashMap.forEach(HashMap.java:1333) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:423) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:119) ~[?:?]
at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:143) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.node.Node.<init>(Node.java:307) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.2.0.jar:7.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.2.0.jar:7.2.0]
... 6 more

unable to load certificate
140650167101328:error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220:
140650167101328:error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509

I don't have the permission to share the output of this command, but it returns successfully.

It might help to say that my pem file is a chain of certificates.

Victor_Guimaraes · July 9, 2019, 1:17pm

Any idea of what might be happening?

ikakavas · July 9, 2019, 1:56pm

No, not really at this point. This looks like the certificate you are using is malformed. Elasticsearch fails to start because it deems the configuration invalid, which in turn is caused by:

Caused by: java.security.cert.CertificateParsingException: signed fields invalid at 
sun.security.x509.X509CertImpl.parse(X509CertImpl.java:1830) ~[?:?] at 
sun.security.x509.X509CertImpl.<init>(X509CertImpl.java:188) ~[?:?] at 
sun.security.provider.X509Factory.parseX509orPKCS7Cert(X509Factory.java:476) ~[?:?] at 
sun.security.provider.X509Factory.engineGenerateCertificates(X509Factory.java:361) ~[?:?] at 
java.security.cert.CertificateFactory.generateCertificates(CertificateFactory.java:478) ~[?:?]

This is not elasticsearch, it's code from sun.security.x509.X509CertImpl that throws this exception ( https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/999dbd4192d0f819cb5224f26e9e7fa75ca6f289/src/java.base/share/classes/sun/security/x509/X509CertImpl.java#L1841) .

Which Java version are you using ?

Victor_Guimaraes · July 9, 2019, 2:57pm

I'm using the Elasticsearch 7.2 embedded jdk.

[root@elasticsearch-gcp-staging bin]# /usr/share/elasticsearch/jdk/bin/java -version
openjdk version "12.0.1" 2019-04-16
OpenJDK Runtime Environment (build 12.0.1+12)
OpenJDK 64-Bit Server VM (build 12.0.1+12, mixed mode, sharing)

I'm using the same certificate on many other places, as the Elasticsearch using SearchGuard and are working fine.

From the link you posted above, the Exception are throw from the method private void parse(DerValue val), my certificate are not a DER, but a PEM certificate as you can see on previous posts. Has any other configurations I must add to my elasticsearch.yml file to force then to parse the certificate as a PEM format?

ikakavas · July 9, 2019, 3:32pm

PEM formatted certificates contain DER encoded data so code eventually needs to parse the DER structure. See this awesome reply from Tim regarding certificate encoding, formatting, etc.

I'm not arguing against that, I'm only saying what your logs are implying.

What Java version are you using in your other Elasticsearch installation?
Can you try and run this Elasticsearch instance with Java 11 ?
Can you try and run your other Elasticsearch installation with Java 12 ?

Not sure what else to suggest without being able to test against your certificate/key.

ikakavas · July 9, 2019, 3:38pm

I was just re-reading this thread and came upon :

Can you please try and create two files by copy-pasting

a) the Leaf certificate from the chain to an certificate.pem file and use that for xpack.security.http.ssl.certificate:
b) The CA certificate from the chain to a ca.pem file and use that for
xpack.security.http.ssl.certificate_authorities:

system · August 6, 2019, 3:38pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.