You set the truststore settings in your Transport client but not your keystore settings as we discussed above.
The truststore settings tell your Transport Client how to know if it can trust the certificate that the Elasticsearch node will present. You need to also set the keystore settings so that your Transport Client knows which key and certificate to use for client authentication.
So if I understand correctly, using :
.put("xpack.security.transport.ssl.keystore.path", "")
.put("xpack.security.transport.ssl.keystore.password", "")
.put("xpack.security.transport.ssl.truststore.path", "")
.put("xpack.security.transport.ssl.truststore.password", "")
Should do the job ?
These are the correct setting keys for SSL yes but the values you use also need to make sense and be correct. The premise here is that you want to achieve TLS mutual authentication and for that to happen your Transport Client needs to trust the certificate Elasticsearch uses, and vice versa , Elasticsearch needs to trust the certificate your Transport Client uses.
In your Transport Client settings: xpack.security.transport.ssl.keystore.path needs to point to a PKCS#12 store that contains a key and certificate that is trusted by Elasticsearch. Your
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
setting in your elasticseach.yml defines what is trusted by Elasticsearch. the elastic-certificates.p12 should contain a CA certificate and Elasticsaerch will trust all the certificates that are signed by that CA certificate.
In your Transport Client settings: xpack.security.transport.ssl.truststore.path needs to point to a PKCS#12 store that contains a CA certificate that determines which certificates your client should trust. It will trust all the certificates that are signed by that CA. And since, in your elasticsearch.yml file you define
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
this means that the Elasticsearch node will use a key and certificate that this PKCS#12 contains for TLS so that certificate needs to be signed by the CA that your Transport Client is configured to use.
Mutual TLS authentication can be tricky to grasp, hope the above helps.
Also, you are again missing settings in your Transport Client settings that are clearly described in our documentation like xpack.security.user, so I'd urge you to go back and read through the documentation once more
Thank you for such a reply ! 
So truststore and keystore can't point to the same .p12 file ?
Because I have the same configuration for http :
truststore and keystore point to the same file which is certs/elastic-certificates.p12 and the ES Hadoop API can connect to the node without any problem ...About the xpack.security.user I tried with that option with the default user but no success, maybe I missed another setting at that time !
It can. A PKCS#12 can contain both a key and certificate that a node can use for TLS as well as a CA certificate that it will use to validate certificates other nodes present. If you use the elasticsearch-certutil cli tool, your PKCS#12 store will contain all the above by default as we describe in our documentation
About the
xpack.security.userI tried with that option with the default user but no success, maybe I missed another setting at that time !
There is no default user. Again, please read through the documentation I keep linking, it will make your effort much much smoother. See step 2 and 3, you need to create a new user and give them the transport_client role and then use that user in your settings.
I'm sorry about the documentation, I read it too quickly.
I added an user named "antoine" with both transport_client and admin roles but I still get the same error. This time the payload has changed :
455300000054000000000000000108004d3603010d417574686f72697a6174696f6e1e4261736963205957353062326c755a54707759584e7a643239795a413d3d0016696e7465726e616c3a7463702f68616e647368616b6500
Once decoded, I can see the username:password.
I'm running out of ideas.
Your Transport Client is still not communicating over TLS.
I added an user named "antoine" with both
transport_clientandadminroles but I still get the same error. This time the payload has changed
this is not relevant to the TLS issue.
Can you please share your code each time, as we can't be sure what you have changed and what is your current configuration.
Assuming you set xpack.security.transport.ssl.keystore.path in your Transport Cleitn as we discused above, can you make sure that the output of
openssl pkcs12 -nocerts -in /path/tothat/keystore
outputs something? No need to share your private key in a post though.
Here is my current code
.put("xpack.security.transport.ssl.enabled", "true")
.put("xpack.security.transport.ssl.keystore.path", "the/path")
.put("xpack.security.transport.ssl.keystore.password", "thepassword")
.put("xpack.security.transport.ssl.truststore.path", "the/path")
.put("xpack.security.transport.ssl.truststore.password", "thepassword")
.put("xpack.security.user", "antoine:password")
The configuration remains the same as discussed (truststore and keystore point to the same .p12)
The command openssl pkcs12 -nocerts -in /path/tothat/keystore does output something (after I enter the 'Import password') but nothing after the PEM pass phrase (I didnt provide any when generating the keystore).
FIY, this keystore is the same I used with the ES Hadoop API so it should be working with the Transport Client I guess.
Can you share a minimum part of your code that we can use to reproduce?
As we established above. the ES Hadoop API uses the http layer and not the transport one and it doesn't need to do client authentication, so it doesn't need to have a key an certificate. The fact that it works for ES Hadoop, only proves that it contains a CA certificate that can validate the certificate that the Elasticsearch node uses for TLS.
Here is the code :
public org.elasticsearch.client.Client create() throws Exception {
org.elasticsearch.common.settings.Settings settings = org.elasticsearch.common.settings.Settings.builder()
.put("cluster.name", "elastic_cluster6")
.put("xpack.security.user", "antoine:password")
.put("xpack.security.transport.ssl.enabled", "true")
.put("xpack.security.transport.ssl.keystore.path", "the/path")
.put("xpack.security.transport.ssl.keystore.password", "thepassword")
.put("xpack.security.transport.ssl.truststore.path", "the/path")
.put("xpack.security.transport.ssl.truststore.password", "thepassword")
.build();
org.elasticsearch.client.transport.TransportClient transportClient = new org.elasticsearch.xpack.client.PreBuiltXPackTransportClient(settings);
transportClient.addTransportAddress(new org.elasticsearch.common.transport.TransportAddress(xxx));
return transportClient;
}
here is the full cluster configuration :
cluster.name: elastic_cluster6
node.name: node-6.4.2
node.master: true
node.data: true
transport.host: 0.0.0.0
transport.tcp.port: 9302
http.port: 9202
network.host: 0.0.0.0
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.type: PKCS12
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.type: PKCS12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.audit.enabled: true
xpack.security.authc.anonymous.roles: superuser
What is the value of xxx in your setup ? Do you pass the port also in there ?
Can you also share a couple of lines (the whole line from the Elasticsearch log, not only the error message) from when your Transport client attempts to communicate to your Elasticsearch node and fails ?
String transport = "ipaddress:9302";
java.util.Map<String, Integer> transportMapping = new java.util.HashMap<>();
String[] transportAddress = transport.split(":");
if (transportAddress .length < 2) {
transportMapping.put(split[0], 9300); //default port if none is provided
} else {
transportMapping.put(split[0], Integer.valueOf(split[1]));
}
for (java.util.Map.Entry<String, Integer> transportAddress : transportMapping.entrySet()) {
transportClient.addTransportAddress(new org.elasticsearch.common.transport.TransportAddress(java.net.InetAddress.getByName(transportAddress.getKey()), transportAddress.getValue()));
}
And here are some of the logs :
[2019-01-15T17:04:05,867][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [node-6.4.2] exception caught on transport layer [NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:9302, remoteAddress=/192.168.150.202:37751}], closing connection
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 455300000054000000000000000108004d3603010d417574686f72697a6174696f6e1e4261736963205957353062326c755a54707759584e7a643239795a413d3d0016696e7465726e616c3a7463702f68616e647368616b6500
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 455300000054000000000000000108004d3603010d417574686f72697a6174696f6e1e4261736963205957353062326c755a54707759584e7a643239795a413d3d0016696e7465726e616c3a7463702f68616e647368616b6500
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1106) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
... 15 more
I also get this message :
None of the configured nodes are available: [{#transport#-1}{h6IgUQwWTDCAQc2wj8HuHA}{ipaddress}{ipaddress:9302}]
Of course, the IP Address is the one provided in the Transport Client (it's not ipaddress which is displayed in the error message)
Not sure what you are doing wrong, but your Tranport Client doesn't use TLS. I assume that the actual code you are using might be different than the minimal part you are sharing, maybe something is different there? For what is worth the following works perfectly talking to an ES node that useing your settings
import org.elasticsearch.action.index.IndexResponse;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.TransportAddress;
import org.elasticsearch.xpack.client.PreBuiltXPackTransportClient;
import static org.elasticsearch.common.xcontent.XContentFactory.*;
import java.net.InetAddress;
import java.util.Date;
public class Client {
public static void main(String[] args) throws Exception{
Settings settings = org.elasticsearch.common.settings.Settings.builder()
.put("cluster.name", "elastic_cluster6")
.put("xpack.security.user", "user:password")
.put("xpack.security.transport.ssl.enabled", "true")
.put("xpack.security.transport.ssl.keystore.path", "/home/ioannis/software/elastic/elasticsearch-6.4.0/config/certs/ks.p12")
.put("xpack.security.transport.ssl.keystore.password", "password")
.put("xpack.security.transport.ssl.truststore.path", "/home/ioannis/software/elastic/elasticsearch-6.4.0/config/certs/ks" +
".p12")
.put("xpack.security.transport.ssl.truststore.password", "password")
.put("xpack.security.transport.ssl.verification_mode", "certificate")
.build();
TransportClient transportClient = new PreBuiltXPackTransportClient(settings);
transportClient.addTransportAddress(new TransportAddress(InetAddress.getByName("localhost"), 9302));
IndexResponse response = transportClient.prepareIndex("twitter", "_doc", "1")
.setSource(jsonBuilder()
.startObject()
.field("user", "kimchy")
.field("postDate", new Date())
.field("message", "trying out Elasticsearch")
.endObject()
)
.get();
System.out.println(response.status().getStatus());
transportClient.close();
}
}
You would either need to share the exact code and settings with us, or work from a simple example like the above and build your code from there. I'm unsure of what more we can do to assist you further with this.
Thanks a lot for the help provided, in fact I can't share the whole code because it is work related.
Now that I know my configuration isn't wrong, I'll look if this is not a network issue or else ?
Again, I'll keep the topic open and will update it as soon as I fix the issue !
Thanks !
Feel free to engage with your Support Engineer via your work support contract, if you have one.
We can't actually be sure of that. I can see only what you share, not what you actually use in your code, so there might be differences.
I can't see how this could be a network issue.
This really looks like your Transport Client is not configured to use TLS.
In fact the other part of the code is non-elastic related, it's only spark code so I don't think it could be the issue. Plus, my transport client works when TLS is disabled 
)
I said so because the code is running on a secured VM, I'm not an expert in networking so I might be wrong about this one 
Here is another part of the code I'm allow to share with you.
It is related to the way we build request :
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
SearchModule searchModule = new SearchModule(Settings.EMPTY, true, Collections.emptyList());
try (XContentParser parser = XContentFactory.xContent(XContentType.JSON)
.createParser(new NamedXContentRegistry(
searchModule.getNamedXContents()),
DeprecationHandler.THROW_UNSUPPORTED_OPERATION,
"{ \"query\": {\"match_all\": {} }}")) //doing that way I can create each query "by hand"
{
searchSourceBuilder.parseXContent(parser);
}
//the client used here is the TransportClient from previous messages
SearchRequestBuilder searchRequestBuilder = new SearchRequestBuilder(client, SearchAction.INSTANCE)
.setTypes("theTypeIUse")
.addSort(FieldSortBuilder.DOC_FIELD_NAME, SortOrder.ASC)
.setScroll(new TimeValue(10000))
.setSearchType(SearchType.DEFAULT);
SearchResponse searchResponse = searchRequestBuilder.setSource(searchSourceBuilder).execute().actionGet();
I know that I am using a deprecated API but I have to deal with it.
When instanciating the SearchModule, maybe I should register the x-pack or transport module instead of Collections.emptyList() but I don't know which parameter to use.
To be honest, I have no idea if this code is "x-pack" ready, but it works when TLS is disabled.
There is nothing you might do after you create the client that could make it stop attempting to connect via TLS with the Elasticsearch node so this part of your code is unrelated. It's very very hard for us to assist you with partial view of what you are actually doing.
I'd try to run the test I sent you against your cluster ( feel free to change the request to something else than indexing random data, it's the initialization of the Client that is of interest) , after you change the files to point to your own PKCS12 and verify this works. Then, try to see what the differences are with your code.
Your code works just fine !
I may have an idea about why mine doesn't.
It appears that the way I build my queries doesn't work when TLS is enabled. I tried with your TransportClient and settings but it doesn't work either.
So I think the TransportClient isn't the problem.
Anyway, I don't really know how to fix my code 
I'm unsure if there is something you could be doing to an already initialized TransportClient that works with TLS to make it stop working with TLS. The Requests are executed by the client, if you keep using the same client object, it would keep working .
In the same manner as before? Seeing plaintext requests in the Elasticsearch node logs or something else?
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.