Having set up TLS for internode communication per
is there a way to confirm that is is being used? E.g. is there something specific that gets written to the log during start up when it's in use, or is there something that can be queried in the cluster to get it to tell you it's in use?
Hi @mikewillis
if you set
xpack.security.transport.ssl.enabled: true
Transport will be on SSL or there will be errors / elasticsearch won't start.
I set transport logging to trace and I do not see an explicit message for SSL and transport.
pretty easy to test just use
openssl s_client -connect hostnameorip:9300
If it is running on SSL you will get the cert and lots of detail if not you won't.
There is no "sort of" running SSL 
for the transport (or any of the elastic ports) if you set SSL enable it will either be SSL or it will not start
After enabled transport SSL the initial result was that the cluster broke. Having worked through reasons why it broke, the cluster now works, and I didn't remove the setting to enable SSL. So the logical inference is that SSL must be working. However I had actually tried the command you suggest before posting and it was partly the result of that which made me ask the question.
[root@bar:production:~]$ grep xpack.security.transport.ssl.enabled /etc/elasticsearch/elasticsearch.yml
xpack.security.transport.ssl.enabled: true
[root@bar:production:~]$ curl -s http://$(hostname):9200;curl -s http://$(hostname):9200/_cluster/health?pretty;
{
"name" : "bar.nowhere.not",
"cluster_name" : "foo",
"cluster_uuid" : "xjUTyk-KS3-KPURhtQ8LAA",
"version" : {
"number" : "8.7.1",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "f229ed3f893a515d590d0f39b05f68913e2d9b53",
"build_date" : "2023-04-27T04:33:42.127815583Z",
"build_snapshot" : false,
"lucene_version" : "9.5.0",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
{
"cluster_name" : "foo",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 1,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
[root@bar:production:~]$ date;openssl s_client -connect $(hostname):9300
Wed 10 May 09:27:54 BST 2023
CONNECTED(00000003)
C041710E737F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:ssl/record/rec_layer_s3.c:308:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 343 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
[root@bar:production:~]$ tail -1 /var/log/elasticsearch/foo.log
[2023-05-10T09:27:54,889][WARN ][o.e.t.TcpTransport ] [bar.nowhere.not] SSL/TLS request received but SSL/TLS is not enabled on this node, got (16,3,1,1), [Netty4TcpChannel{localAddress=/10.70.13.199:9300, remoteAddress=/10.70.13.199:44608, profile=default}], closing connection
[root@bar:production:~]$
Hi @mikewillis
The http and transport are completely separate interfaces and are completely independent with respect to SSL.
Please post your entire elasticsearch.yml
How did you install / start / stop elastic ... if by package have you reloaded the daemon
sudo /bin/systemctl daemon-reload
Is that the only log line for transport if you enabled DEBUG or TRACE there should be more? Did the node start?
logger.org.elasticsearch.transport: TRACE
Is this one of many nodes? Are All nodes configured the same?
I've figured out why no certificates were being returned from 9300. The
elasticsearch.yml contained the following xpack related config:
xpack.security.enabled: false
xpack.security.audit.enabled: false
xpack.ml.enabled: false
xpack.watcher.enabled: false
xpack.graph.enabled: false
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: foo-certificates.p12
xpack.security.transport.ssl.truststore.path: foo-certificates.p12
xpack.security.http.ssl.enabled: false
Note the first line of that!
xpack.security.enabled was false due to the configuration I'm using having initially been used with earlier versions of Elasticsearch where the security stuff cost money and we didn't need it. And I'd forgotten it was there, far enough away from the newly added xpack.security.transport.ssl settings in the file that it took me a while to notice it. Having set xpack.security.enabled to true if I use openssl s_client to connect to 9300 I get expected output including the certificate.
Whilst my problem was obviously largely self inflicted I think it's worth me explaining what I did as it feels like Elasticsearch has behaved somewhat unhelpfully.
I installed Elasticsearch on all three nodes from the rpm using dnf module of Ansible and Ansible. Then I put in place configuration with
xpack.security.transport.ssl.enabled: false
xpack.security.http.ssl.enabled: false
for simplicity just to check I could get a working cluster. Then I reinstalled it all with
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: false
and followed
to create a certificate without a password (for simplicity while experimenting). Having followed that Elasticsearch refused to start with an error about not being able to read foo-certificates.p12 which turned out to be because I'd put the file in /usr/share/elasticsearch/ by mistake. I moved it to /etc/elasticsearch/ and Elasticsearch still didn't start because of filesystem permissions. I fixed that and then got:
Caused by: org.elasticsearch.common.ssl.SslConfigException: cannot read
configured [PKCS12] keystore (as a truststore)
[/etc/elasticsearch/foo-certificates.p12] - this is usually caused by an
incorrect password; (a keystore password was provided)
Which was confusing because I had not set a password, because my certificate didn't have one.
While investigating that I discovered the rpm has post install scripts which create a certificate with a password and a keystore with the password in. (All the output of the rpm post install scripts which is visible when you install the rpm manually with rpm or dnf, and tells you about a certificate being created and a password set for the elastic user, is not visible, at least by default, when you install with Ansible dnf module.) The documentation I followed mentions the need to add a password for your certificate to the keystore, if you set one, but it doesn't say anything about how a password will already be in the keystore if you installed with the rpm and Elasticsearch will try to use that with a certificate you made yourself that doesn't have a password.
To get Elasticsearch to use foo-certificates.p12 I deleted the keystores on all the nodes and then created empty ones. I have repeated this whole process several times to confirm my findings with consistent results.
So after setting
xpack.security.transport.ssl.enabled: true
I had to fix several issues that prevented Elasticsearch from starting because of problems relating to transport SSL configuration. Having fixed those issues and getting a working cluster it seems reasonable to assume that transport SSL would be being used. If Elasticsearch isn't trying to use transport SSL then why refuse to start because it can't read a certificate for transport SSL. But evidently it is possible to give Elasticsearch configuration which means it will refuse to start
because of problems with transport SSL, but won't use transport SSL once those problems are fixed.
The issue I have now is that with xpack.security.enabled being true authentication is in effect on the REST API. All I wanted to do was enable transport SSL. I can't see a way to get transport SSL without the REST API authentication.
Hi @mikewillis
1st glad you found it but sorry it was such an adventure.
A couple of thoughts and hindsight is 20/20
The reason I asked and it is generally a good practice in a post is to post the entire .yml as we probably would have spotted that inconsistency right away.
No ... the progression of security on elasticsearch is (and has been this way for many years)
As to all the combinations you tried ... a bit hard to track... but in 8.X and going forward security is enabled by default which is definitely by design.. when you turn it off then you would be expected to follow the manual process in detail / per the docs.
Could there be better error messages ... perhaps... we are always open to feedback and welcome issues / comments and feature suggestions against the elasticsearch repo.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.