"io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record" after ssl enabling my cluster

Elastic Stack Elasticsearch
1

Hello

I'm testing X-Pack security and I wanted to encrypt communication in my cluster (non-production).
So I followed the instructions here and created self signed certificate for each node. I also changed/added all needed parameters in elasticsearch configuration file on each node.

Now, when I restart the cluster the nodes don't seem to talk to each other and I find this kind of errors in the log: "io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record".
I googled it and understood that a client is trying to access a SSL enabled node via non-ssl connection. I double checked all nodes and they all configured to use ssl.

I even shut down all nodes except one, so there should'nt be any incoming traffic, but still I see the same error again and again in its log !!! What can be the reason ? what am I doing wrong ?

(If it matters, I am using a trial license, which I think is good for 30 days)

Thanks

Guy

2

Hello,

When opening an issue like this, it is important to provide log output, configuration, and versions of the product. Once that has been provided, we will be able to assist you with this.

3

Same issue for me

The keys were generated from the sec_course.zip
all paths, names and ip addresses to setup the keys are correct and Ive done the exercise twice from the beginning to be sure
...
CODE

xpack:
    ssl:
        key: /usr/share/elasticsearch/bin/config/x-pack/nn775/nn775.key
        certificate: /usr/share/elasticsearch/bin/config/x-pack/nn775/nn775.crt
        # This points to where the CA cert is - every node shares this

certificate_authorities: /usr/share/elasticsearch/bin/config/ca/ca.crt

# We need to ensure that the transport (port 9300) and HTTP (port
# 9200) protocols secure communication is enabled:
xpack:
    security:
        transport.ssl.enabled: true
        http.ssl.enabled: true

...
CODE
...

log is here
bottom of page

https://sites.google.com/site/developtroubleshooting/home/logs

https://sites.google.com/site/developtroubleshooting/home/logs/not%20%20a%20tls%20record.txt?attredirects=0&d=1

4

This is HTTP traffic o.e.x.s.t.n.SecurityNetty4HttpServerTransport. What other settings do you have? What else is trying to connect using HTTP?

5

Im setting this up - per the course

I'm guessing this will protect the 9200-9300 ports of elasticsearch.
This would be from either this
https://nn775:9200/sales_record/_search?pretty

or kibana tryining to connect to elastic

this is my elsticsearch.yml file

...
CODE
....
xpack:
ssl:
key: /usr/share/elasticsearch/bin/config/x-pack/nn775/nn775.key
certificate: /usr/share/elasticsearch/bin/config/x-pack/nn775/nn775.crt
# This points to where the CA cert is - every node shares this
certificate_authorities: /usr/share/elasticsearch/bin/config/ca/ca.crt

    # We need to ensure that the transport (port 9300) and HTTP (port
    # 9200) protocols secure communication is enabled:
    xpack:
        security:
            transport.ssl.enabled: true
            http.ssl.enabled: true

    # AUDIT ########################################################
    xpack.security.audit.enabled: true
    xpack.security.audit.outputs: [ index, logfile ]
    xpack.security.audit.index.settings:
      index:
        number_of_shards: 1
        number_of_replicas: 1

    xpack.security.audit.logfile.events.exclude: [ access_granted ]
    xpack.security.audit.index.events.exclude: [ access_granted ]

    ###################################################################
    #xpack.monitoring.enabled: false
    #Set to false to disable X-Pack monitoring. Configure in both elasticsearch.yml and kibana.yml.

    #xpack.graph.enabled: false
    #Set to false to disable X-Pack graph. Configure in both elasticsearch.yml and kibana.yml.

    #xpack.watcher.enabled: false
    #Set to false to disable Watcher. Configure in elasticsearch.yml only.

    #xpack.reporting.enabled: false
    #Set to false to disable X-Pack reporting. Configure in kibana.yml only.

    bootstrap.system_call_filter: false
    cluster.name: EScluster-rack2
    node.name: ${HOSTNAME}
    node.master: true
    node.data: true
    network.host: ${HOSTNAME}
    http.port: 9200
    discovery.zen.minimum_master_nodes: 3
    path.data: /mnt/data/elk/5.2.data
    path.logs: /var/log/elasticsearch
    #
    # Make sure that the `ES_HEAP_SIZE` environment variable is set to about half the memory
    # available on the system and that the owner of the process is allowed to use this limit.
    #
    discovery.zen.ping.unicast.hosts: ["nn775","sn776","dn777","dn778","dn779"]
    #
    # node.max_local_storage_nodes: 1
    # Require explicit names when deleting indices:
    # action.destructive_requires_name: true

...
CODE
...

I wasnt getting these errors before I added the certs to the elasticsearc.xml

6

Sounds like this could be the source. Did you change kibana to use https and trust the cert?

7

This is from the security course- lab 4 getting up ttls

KIbana was the next step - let me set that and see...

8

same error in teh es.log

Ive added below to the kibana.yml

elasticsearch.url: "https://192.168.1.180:9200"
elasticsearch.ssl.verify: false 

console.proxyConfig:
 - match:
    host: "*"
    port: "{9200..9202}"

   ssl:
    ca: "/usr/share/elasticsearch/bin/config/ca/ca.crt"

restarted both kibana and es -

when I try this
https://nn775:9200/sales_record/_search?pretty

I get the expected behavior of this site cant be trusted - procede anyway?
I do and am prompted to login.

Ive reset the password properly for elastic user and its not accepting the credentials
ES log is still churning out this error not a SSL/TLS record

9

I removed the security settings and everything is working.

Ill try again - do you have access to the security course? chapter 4
I beleive I followed it to the letter. Up until this everything was very accurate in the course

10

so I regenerated the certs and keys again and I have ES working ok with teh certs - no errors and it blocks the https://nn775:9200

Im getting errors to this effect

[2017-03-21T16:45:33,922][INFO ][o.e.x.m.e.Exporters      ] [nn775] skipping exporter [default_local] as it is not ready yet
[2017-03-21T16:45:43,926][INFO ][o.e.x.m.e.Exporters      ] [nn775] skipping exporter [default_local] as it is not ready yet
[2017-03-21T16:45:53,929][INFO ][o.e.x.m.e.Exporters      ] [nn775] skipping exporter [default_local] as it is not ready yet
[2017-03-21T16:46:03,933][INFO ][o.e.x.m.e.Exporters      ] [nn775] skipping exporter [default_local] as it is not ready yet

Ive made this entry to my kibana.yml
elasticsearch.url: "https://192.168.1.180:9200"
elasticsearch.ssl.verify: false

console.proxyConfig:
 - match:
    host: "*"
    port: "{9200..9202}"

   ssl:
    ca: "/usr/share/elasticsearch/bin/config/ca/ca.crt"

browser says website cant be reached...

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.