Cannot connect Java client on ES 6.2.1 with Xpack

[ivanqwam (Ivan Monnier)]

I guess that you got this post a thousand times but I could not find my exact issue on the forum. I apologize if it is a duplicate.

Trying to set up Xpack on ES 6.2.1, I did the following

1) On master node
cd /usr/software/elasticsearch
bin/elasticsearch-plugin install x-pack
mkdir config/cert
bin/x-pack/certutil ca
mv elastic-stack-ca.p12 config/cert/
bin/x-pack/certutil cert --ca conf/cert/elastic-stack-ca.p12
mv elastic-certificates.p12 config/cert/
restartelasticsearch.sh

2) On all other nodes
cd /usr/software/elasticsearch
bin/elasticsearch-plugin install x-pack
mkdir config/cert
scp master:/usr/software/elasticsearch/config/cert/elastic-stack-ca.p12 config/cert/.
bin/x-pack/certutil cert --ca conf/cert/elastic-stack-ca.p12
mv elastic-certificates.p12 config/cert/
restartelasticsearch.sh

3) On master node
bin/x-pack/setup-passwords interactive

4) On kibana
cd /usr/software/kibana
bin/kibana-plugin install x-pack
vi config/kibana.yml
added:
elasticsearch.username: "kibana"
elasticsearch.password: "XXXXXX"
restartkibana.sh

5) On master node
cd /usr/software/elasticsearch
bin/x-pack/users useradd mysuperuser -p XXXXXXX -r superuser

As you can see, it is pretty straightforward with the documentation examples.
At this stage, everything is working fine. All nodes are connected, Kibana can connect.

Trying to add the java client in the dance:

6) On kibana, connected as super user
Created user myjavauser with transport_client role

7) On Java
Modified the client to be the Xpack client
Performed the maven dependencies fetch.
Configured login, password and paths to certificates

Here is my code

ses.log(null, "DEBUG", "QESHost_elastic", "connect()", 3, "Connection avec Xpack");
ses.log(null, "DEBUG", "QESHost_elastic", "connect()", 3, "Login", login);
ses.log(null, "DEBUG", "QESHost_elastic", "connect()", 3, "Path certificat", pathCertificate);
ses.log(null, "DEBUG", "QESHost_elastic", "connect()", 3, "Path authorities", pathAuthorities);

Settings settings = null;
if (!pathCertificate.isEmpty()) {
    ses.log(null, "DEBUG", "QESHost_elastic", "connect()", 3, "Avec certificats");
    settings = Settings.builder()
        .put("cluster.name", cluster)
        .put("xpack.security.user", login + ":" + password)
        .put("xpack.ssl.certificate", pathCertificate)
        .put("xpack.ssl.certificate_authorities", pathAuthorities)
        .put("xpack.security.transport.ssl.enabled", "true")
        .put("xpack.security.transport.ssl.verification_mode", "certificate")
        .build();
        client = new PreBuiltXPackTransportClient(settings).addTransportAddress(new 
                                                TransportAddress(InetAddress.getByName(host), port));
   }

Here is the run:

2018/02/28 10:35:48 568-myapp-null-TRACE-QESHost_elastic-sessionClient()	: Nouvelle connexion ...:localhost:9300|
2018/02/28 10:35:48 568-myapp-null-DEBUG-QESHost_elastic-connect()	: cluster:elasticsearch|
2018/02/28 10:35:48 568-myapp-null-DEBUG-QESHost_elastic-connect()	: host:localhost|
2018/02/28 10:35:48 568-myapp-null-DEBUG-QESHost_elastic-connect()	: port:9300|
2018/02/28 10:35:48 568-myapp-null-DEBUG-QESHost_elastic-connect()	: Connection avec Xpack|
2018/02/28 10:35:48 568-myapp-null-DEBUG-QESHost_elastic-connect()	: Login:myjavauser|
2018/02/28 10:35:48 568-myapp-null-DEBUG-QESHost_elastic-connect()	: Path certificat:/usr/software/elasticsearch/config/cert/elastic-certificates.p12|
2018/02/28 10:35:48 568-myapp-null-DEBUG-QESHost_elastic-connect()	: Path authorities:/usr/software/elasticsearch/config/cert/elastic-certificates.p12|
2018/02/28 10:35:48 568-myapp-null-DEBUG-QESHost_elastic-connect()	: Avec certificats|
2018/02/28 10:35:49 686-myapp-null-ERROR-QESHost_elastic-connect()	: Erreur de connexion:ElasticsearchException[failed to initialize a TrustManagerFactory]; nested: MalformedInputException[Input length = 1];|
java.lang.RuntimeException: ElasticsearchException[failed to initialize a TrustManagerFactory]; nested: MalformedInputException[Input length = 1];
	at org.elasticsearch.xpack.core.XPackClientPlugin.getTransports(XPackClientPlugin.java:385)
	at org.elasticsearch.common.network.NetworkModule.<init>(NetworkModule.java:126)
	at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:177)
	at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:273)
	at org.elasticsearch.transport.client.PreBuiltTransportClient.<init>(PreBuiltTransportClient.java:128)
	at org.elasticsearch.xpack.client.PreBuiltXPackTransportClient.<init>(PreBuiltXPackTransportClient.java:59)
...
	at com.qwam.qesutil.qeselasticindex.main(qeselasticindex.java:610)
Caused by: ElasticsearchException[failed to initialize a TrustManagerFactory]; nested: MalformedInputException[Input length = 1];
	at org.elasticsearch.xpack.core.ssl.PEMTrustConfig.createTrustManager(PEMTrustConfig.java:60)
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:412)
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:448)
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:91)
	at org.elasticsearch.xpack.core.XPackClientPlugin.getTransports(XPackClientPlugin.java:383)
	... 12 more
Caused by: java.nio.charset.MalformedInputException: Input length = 1
	at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
	at java.io.InputStreamReader.read(InputStreamReader.java:184)
	at java.io.BufferedReader.read1(BufferedReader.java:210)
	at java.io.BufferedReader.read(BufferedReader.java:286)
	at java.io.BufferedReader.fill(BufferedReader.java:161)
	at java.io.BufferedReader.readLine(BufferedReader.java:324)
	at java.io.BufferedReader.readLine(BufferedReader.java:389)
	at org.bouncycastle.util.io.pem.PemReader.readPemObject(Unknown Source)
	at org.bouncycastle.openssl.PEMParser.readObject(Unknown Source)
	at org.elasticsearch.xpack.core.ssl.CertUtils.readCertificates(CertUtils.java:328)
	at org.elasticsearch.xpack.core.ssl.CertUtils.readCertificates(CertUtils.java:315)
	at org.elasticsearch.xpack.core.ssl.CertUtils.readCertificates(CertUtils.java:307)
	at org.elasticsearch.xpack.core.ssl.PEMTrustConfig.createTrustManager(PEMTrustConfig.java:57)
	... 16 more

I cannot find the reason of this exception:

Caused by: java.nio.charset.MalformedInputException: Input length = 1

I am missing something there for sure.
Can You help me ?

[TimV (Tim Vernum)]

You are using a .p12 (PKCS#12) file for xpack.ssl.certificate and xpack.ssl.certificate_authorities, but those settings expect PEM files instead.

Both PKCS#12 and PEM are formats for storing TLS certificates and keys, but they are very different and require different settings.

You want something like:

        .put("xpack.ssl.keystore.path", pathCertificate)
        .put("xpack.ssl.truststore.path", pathAuthorities)

[ivanqwam (Ivan Monnier)]

Thank you Tim, this did the trick. It's solved.

May I kindly suggest that you update this documentation page:

https://www.elastic.co/guide/en/x-pack/current/java-clients.html

The whole process describing migration to Xpack gives example with unsigned certificates (PKCS#12) .
And in this page, all the sudden, it is talking about signed certificate from an authority (PEM).

Keys are different for each type of certificate. If you did not have told me the correct keys, I did not have many hints to guess them

Certificates noobs like me get pretty easily lost with the current documentation, on this specific point.

Anyway the documentation is improving very fast those days, it is only a suggestion.

[TimV (Tim Vernum)]

Thanks, I've raised an internal issue to get those docs updated.