# Verify internode communication is using TLS **URL:** \ **Category:** Elasticsearch\ **Created:** [May 9, 2023, 4:15pm UTC](https://discuss.elastic.co/t/verify-internode-communication-is-using-tls/332975 "2023-05-09T16:15:54Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![mikewillis](https://avatars.discourse-cdn.com/v4/letter/m/b2d939/32.png) [@mikewillis](https://discuss.elastic.co/u/mikewillis)\ **Post date:** [May 9, 2023, 4:15pm UTC](https://discuss.elastic.co/t/verify-internode-communication-is-using-tls/332975/1 "2023-05-09T16:15:54Z")
Having set up TLS for internode communication per > **[Set up basic security for the Elastic Stack | Elasticsearch Guide \[8.7\] |...](https://www.elastic.co/guide/en/elasticsearch/reference/8.7/security-basic-setup.html)** is there a way to confirm that is is being used? E.g. is there something specific that gets written to the log during start up when it's in use, or is there something that can be queried in the cluster to get it to tell you it's in use? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 10, 2023, 3:54am UTC](https://discuss.elastic.co/t/verify-internode-communication-is-using-tls/332975/2 "2023-05-10T03:54:53Z")
Hi @mikewillis if you set `xpack.security.transport.ssl.enabled: true` Transport will be on SSL or there will be errors / elasticsearch won't start. I set transport logging to trace and I do not see an explicit message for SSL and transport. pretty easy to test just use `openssl s_client -connect hostnameorip:9300` If it is running on SSL you will get the cert and lots of detail if not you won't. There is no "sort of" running SSL 🙂 for the transport (or any of the elastic ports) if you set SSL enable it will either be SSL or it will not start ---
**Author:** ![mikewillis](https://avatars.discourse-cdn.com/v4/letter/m/b2d939/32.png) [@mikewillis](https://discuss.elastic.co/u/mikewillis)\ **Post date:** [May 10, 2023, 8:39am UTC](https://discuss.elastic.co/t/verify-internode-communication-is-using-tls/332975/3 "2023-05-10T08:39:27Z")
After enabled transport SSL the initial result was that the cluster broke. Having worked through reasons why it broke, the cluster now works, and I didn't remove the setting to enable SSL. So the logical inference is that SSL must be working. 🙂 However I had actually tried the command you suggest before posting and it was partly the result of that which made me ask the question. ```auto [root@bar:production:~]$ grep xpack.security.transport.ssl.enabled /etc/elasticsearch/elasticsearch.yml xpack.security.transport.ssl.enabled: true [root@bar:production:~]$ curl -s http://$(hostname):9200;curl -s http://$(hostname):9200/_cluster/health?pretty; { "name" : "bar.nowhere.not", "cluster_name" : "foo", "cluster_uuid" : "xjUTyk-KS3-KPURhtQ8LAA", "version" : { "number" : "8.7.1", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "f229ed3f893a515d590d0f39b05f68913e2d9b53", "build_date" : "2023-04-27T04:33:42.127815583Z", "build_snapshot" : false, "lucene_version" : "9.5.0", "minimum_wire_compatibility_version" : "7.17.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "You Know, for Search" } { "cluster_name" : "foo", "status" : "green", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 1, "active_shards" : 2, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } [root@bar:production:~]$ date;openssl s_client -connect $(hostname):9300 Wed 10 May 09:27:54 BST 2023 CONNECTED(00000003) C041710E737F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:ssl/record/rec_layer_s3.c:308: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 0 bytes and written 343 bytes Verification: OK --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) --- [root@bar:production:~]$ tail -1 /var/log/elasticsearch/foo.log [2023-05-10T09:27:54,889][WARN][o.e.t.TcpTransport] [bar.nowhere.not] SSL/TLS request received but SSL/TLS is not enabled on this node, got (16,3,1,1), [Netty4TcpChannel{localAddress=/10.70.13.199:9300, remoteAddress=/10.70.13.199:44608, profile=default}], closing connection [root@bar:production:~]$ ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 10, 2023, 1:47pm UTC](https://discuss.elastic.co/t/verify-internode-communication-is-using-tls/332975/4 "2023-05-10T13:47:50Z")
Hi @mikewillis The http and transport are completely separate interfaces and are completely independent with respect to SSL. Please post your entire elasticsearch.yml How did you install / start / stop elastic ... if by package have you reloaded the daemon `sudo /bin/systemctl daemon-reload` Is that the only log line for transport if you enabled DEBUG or TRACE there should be more? Did the node start? `logger.org.elasticsearch.transport: TRACE` Is this one of many nodes? Are All nodes configured the same? ---
**Author:** ![mikewillis](https://avatars.discourse-cdn.com/v4/letter/m/b2d939/32.png) [@mikewillis](https://discuss.elastic.co/u/mikewillis)\ **Post date:** [May 12, 2023, 1:40pm UTC](https://discuss.elastic.co/t/verify-internode-communication-is-using-tls/332975/5 "2023-05-12T13:40:48Z")
I've figured out why no certificates were being returned from `9300`. The `elasticsearch.yml` contained the following xpack related config: ```auto xpack.security.enabled: false xpack.security.audit.enabled: false xpack.ml.enabled: false xpack.watcher.enabled: false xpack.graph.enabled: false xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.client_authentication: required xpack.security.transport.ssl.keystore.path: foo-certificates.p12 xpack.security.transport.ssl.truststore.path: foo-certificates.p12 xpack.security.http.ssl.enabled: false ``` Note the first line of that! `xpack.security.enabled` was `false` due to the configuration I'm using having initially been used with earlier versions of Elasticsearch where the security stuff cost money and we didn't need it. And I'd forgotten it was there, far enough away from the newly added `xpack.security.transport.ssl` settings in the file that it took me a while to notice it. Having set `xpack.security.enabled` to `true` if I use `openssl s_client` to connect to `9300` I get expected output including the certificate. Whilst my problem was obviously largely self inflicted I think it's worth me explaining what I did as it feels like Elasticsearch has behaved somewhat unhelpfully. I installed Elasticsearch on all three nodes from the rpm using `dnf` module of Ansible and Ansible. Then I put in place configuration with ```auto xpack.security.transport.ssl.enabled: false xpack.security.http.ssl.enabled: false ``` for simplicity just to check I could get a working cluster. Then I reinstalled it all with ```auto xpack.security.transport.ssl.enabled: true xpack.security.http.ssl.enabled: false ``` and followed > **[Set up basic security for the Elastic Stack | Elasticsearch Guide \[8.7\] |...](https://www.elastic.co/guide/en/elasticsearch/reference/8.7/security-basic-setup.html)** to create a certificate without a password (for simplicity while experimenting). Having followed that Elasticsearch refused to start with an error about not being able to read `foo-certificates.p12` which turned out to be because I'd put the file in `/usr/share/elasticsearch/` by mistake. I moved it to `/etc/elasticsearch/` and Elasticsearch still didn't start because of filesystem permissions. I fixed that and then got: ```auto Caused by: org.elasticsearch.common.ssl.SslConfigException: cannot read configured [PKCS12] keystore (as a truststore) [/etc/elasticsearch/foo-certificates.p12] - this is usually caused by an incorrect password; (a keystore password was provided) ``` Which was confusing because I had not set a password, because my certificate didn't have one. While investigating that I discovered the rpm has post install scripts which create a certificate with a password and a keystore with the password in. (All the output of the rpm post install scripts which is visible when you install the rpm manually with `rpm` or `dnf`, and tells you about a certificate being created and a password set for the `elastic` user, is not visible, at least by default, when you install with Ansible `dnf` module.) The documentation I followed mentions the need to add a password for your certificate to the keystore, if you set one, but it doesn't say anything about how a password will already be in the keystore if you installed with the rpm and Elasticsearch will try to use that with a certificate you made yourself that doesn't have a password. To get Elasticsearch to use `foo-certificates.p12` I deleted the keystores on all the nodes and then created empty ones. I have repeated this whole process several times to confirm my findings with consistent results. So after setting ```auto xpack.security.transport.ssl.enabled: true ``` I had to fix several issues that prevented Elasticsearch from starting because of problems relating to transport SSL configuration. Having fixed those issues and getting a working cluster it seems reasonable to assume that transport SSL would be being used. If Elasticsearch isn't trying to use transport SSL then why refuse to start because it can't read a certificate for transport SSL. But evidently it is possible to give Elasticsearch configuration which means it will refuse to start because of problems with transport SSL, but won't use transport SSL once those problems are fixed. The issue I have now is that with `xpack.security.enabled` being `true` authentication is in effect on the REST API. All I wanted to do was enable transport SSL. I can't see a way to get transport SSL without the REST API authentication. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 12, 2023, 2:51pm UTC](https://discuss.elastic.co/t/verify-internode-communication-is-using-tls/332975/6 "2023-05-12T14:51:18Z")
Hi @mikewillis 1st glad you found it but sorry it was such an adventure. A couple of thoughts and hindsight is 20/20 The reason I asked and it is generally a good practice in a post is to post the entire .yml as we probably would have spotted that inconsistency right away. > [@mikewillis](#): > > The issue I have now is that with `xpack.security.enabled` being `true` authentication is in effect on the REST API. All I wanted to do was enable transport SSL. I can't see a [https://www.elastic.co/guide/en/elasticsearch/reference/8.7/security-basic-setup.htmlway](https://www.elastic.co/guide/en/elasticsearch/reference/8.7/security-basic-setup.htmlway) to get transport SSL without the REST API authentication. No ... the progression of security on elasticsearch is (and has been this way for many years) 1. Enable security which enables / forces authentication on the HTTP REST endpoint 2. Enable transport layer security 3. Enable HTTP interface security As to all the combinations you tried ... a bit hard to track... but in 8.X and going forward security is enabled by default which is definitely by design.. when you turn it off then you would be expected to follow the manual process in detail / per the docs. Could there be better error messages ... perhaps... we are always open to feedback and welcome issues / comments and feature suggestions against the elasticsearch repo. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 9, 2023, 2:51pm UTC](https://discuss.elastic.co/t/verify-internode-communication-is-using-tls/332975/7 "2023-06-09T14:51:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.