Setting up a PKI realm using PEM certificates

Elastic Stack Elasticsearch
1

Hello,

I am trying to set up a PKI realm for authentication and I am having trouble with this. I am using elastic search 5.3.0 and I set up a simple cluster of one node hosted on localhost. My purpose is to authenticate the clients (e-g, using the console with cURL to access the DB) with a PEM certificate.

To do so, I carefully followed the documentation here https://www.elastic.co/guide/en/x-pack/current/pki-realm.html and here https://www.elastic.co/guide/en/x-pack/current/ssl-tls.html .

I used certgen to create my PEM certificates : I generated a (certificate,key) for my one and only node and another (certificate,key) couple for the client. It also generated the CA certificate authority ca.cert. I put the node certificate and the CA certificate in the config/x-pack/ folder.

Then, when I try to run bin/elasticsearch, I am getting a lot of errors. I can't even try the authentication process because I do not manage to enable tls in elastic search. Here are my errors

console error output 
[2017-04-20T12:20:12,652][WARN ][o.e.x.s.t.n.SecurityNetty4Transport] [main_node] exception caught on transport layer [[id: 0xc9dc8cf5, L:0.0.0.0/0.0.0.0:9300 ! R:/0:0:0:0:0:0:0:1:45487]], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
	[...]
Caused by: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
[...]

[2017-04-20T12:20:12,652][WARN ][o.e.x.s.t.n.SecurityNetty4Transport] [main_node] exception caught on transport layer [[id: 0x87d68d4e, L:0.0.0.0/0.0.0.0:45487 ! R:/0:0:0:0:0:0:0:1:9300]], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
	[...]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
[...]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
[...]

Caused by: java.security.cert.CertificateException: No subject alternative names present
[...]

[2017-04-20T12:20:13,504][WARN ][o.e.x.s.t.n.SecurityNetty4Transport] [main_node] exception caught on transport layer [[id: 0xd0de3aef, L:0.0.0.0/0.0.0.0:9300 ! R:/0:0:0:0:0:0:0:1:45493]], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
[...]
Caused by: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
[...]

[2017-04-20T12:20:13,504][WARN ][o.e.x.s.t.n.SecurityNetty4Transport] [main_node] exception caught on transport layer [[id: 0x838e0c17, L:0.0.0.0/0.0.0.0:45493 ! R:/0:0:0:0:0:0:0:1:9300]], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
	[...]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
[...]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
[...]
Caused by: java.security.cert.CertificateException: No subject alternative names present
[...]

[2017-04-20T12:20:14,504][WARN ][o.e.x.s.t.n.SecurityNetty4Transport] [main_node] exception caught on transport layer [[id: 0x21dab67d, L:0.0.0.0/0.0.0.0:9300 ! R:/0:0:0:0:0:0:0:1:45505]], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
	[...]
Caused by: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
[...]

[2017-04-20T12:20:14,504][WARN ][o.e.x.s.t.n.SecurityNetty4Transport] [main_node] exception caught on transport layer [[id: 0x2211c6d4, L:0.0.0.0/0.0.0.0:45505 ! R:/0:0:0:0:0:0:0:1:9300]], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
[...]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
[...]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
[...]
Caused by: java.security.cert.CertificateException: No subject alternative names present
[...]

and here is my elasticsearch.yml

elasticsearch.yml 
cluster.name: my-application

node.name: main_node

#Enabling TLS
xpack.ssl.key: /home/me/elasticsearch-5.3.0/config/x-pack/main_node/main_node.key
xpack.ssl.certificate: /home/me/elasticsearch-5.3.0/config/x-pack/main_node/main_node.crt
xpack.ssl.certificate_authorities: [ "/home/me/elasticsearch-5.3.0/config/x-pack/ca/ca.crt" ]

#tls
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true

#enabling PKI
xpack:
  security:
    authc:
      realms:
        pki1:
          type: pki

I am using: openjdk version "1.8.0_111", x-pack 5.3, elastic search 5.3.0

I checked the troubleshooting page https://www.elastic.co/guide/en/x-pack/current/security-troubleshooting.html for these exceptions:

  • No subject alternative names present: When creating the certificates with certgen, I did not specify any ip nor DNS. What I understood from the doc is that it was not required and it would skip the hostname verification. I also tried to create the certificate with ip : 127.0.0.1 (for localhost) and it did not work neither.

  • unkown certificate: I have no clue why my certificates are not recognized. Is it trying to find my CA cert in the Java truststore ? I did define xpack.ssl.certificate_authorities in elasticsearch.yml to use the PEM file generated from certgen.

So, what am I doing wrong here ?

Thank you !

2

Did you use a single CA for both the node-TLS cert and client-PKI certs? Or run them under 2 different CAs?

Received fatal alert: certificate_unknown

This message typically indicates that the client doesn't trust the server certificate, and forcibly closed the connection. At a guess, it looks like your cURL client isn't configured to trust the sever's CA/cert.

3

I used only one CA, for both TLS and client-PKI. (I created both client cert and node cert in one pass of certgen, it gave me a single CA file)

I am getting the javax.net.ssl.SSLException: certificate unkwown. So, according to the troubleshooting documentation, it is a server side exception. Added to that, I get these errors when I run bin/elasticsearch and I believe it does not involve cURL at all. And I also tried to use a keystore and a truststore instead of PEM certificates, but I get the exact same errors. I have the feeling that x-pack is trying to use the default Java keystore (even if I defined the location of the PEM files), would that be a possible cause ?

Thanks for your help !

4

Yes, it is a server side exception, that is typically reported when the client forcibly closes the connection because it did not trust the server's certificate.

In this case, I misread your logs and took it to be an HTTP exception, but on closer inspection it is actually node-to-node communication that is failing - I was confused because you say you only have 1 node in your cluster, but these logs clearly indicate that something is trying to connect to the "transport" port.

Do you have another node/cluster running on the same machine? Or are you trying to connect to this node using TransportClient?

Either some other node is trying to connect to your cluster, or your cluster is trying to connect to some other nodes (or both) and the TLS configurations don't match. The details of the exceptions aren't particularly important until we can work out where that TCP traffic is coming from.

5

Well, I did further investigation and I am still unable to find the problem. I did not find what could be interacting with the node. I do not have an other cluster nor an other node on this machine. I am not using TransportClient. I fully deleted my ES installation and I reinstalled it. I tried to change the name of the cluster and the node to avoid potential conflicts. I tried to change the ports too... nothing worked. When I do not enable TLS and I request the nodes of my cluster, there is only one node and here is the response:

curl -XGET 'http://localhost:9200/_nodes?pretty' (TLS and auth disabled) 
{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "my-pki-realm",
  "nodes" : {
    "rv5V5LFYS_6M8igSJ0LyOg" : {
      "name" : "main_node",
      "transport_address" : "127.0.0.1:9300",
      "host" : "127.0.0.1",
      "ip" : "127.0.0.1",
      "version" : "5.3.0",
      "build_hash" : "3adb13b",
      "total_indexing_buffer" : 207775334,
      "roles" : [
        "master",
        "data",
        "ingest"
      ],
      "settings" : {
        "cluster" : {
          "name" : "my-pki-realm"
        },
        "node" : {
          "name" : "main_node"
        },
        "path" : {
          "logs" : "/home/me/elasticsearch-5.3.0/logs",
          "home" : "/home/me/elasticsearch-5.3.0"
        },
        "client" : {
          "type" : "node"
        },
        "http" : {
          "type" : "security4"
        },
        "transport" : {
          "type" : "security4"
        },
        "xpack" : {
          "ssl" : {
            "certificate_authorities" : [
              "/home/me/elasticsearch-5.3.0/config/x-pack/ca/ca.crt"
            ]
          },
          "security" : {
            "http" : {
              "ssl" : {
                "enabled" : "false"
              }
            },
            "transport" : {
              "ssl" : {
                "enabled" : "false"
              }
            }
          }
        }
      },
      "os" : {
        "refresh_interval_in_millis" : 1000,
        "name" : "Linux",
        "arch" : "amd64",
        "version" : "3.19.0-32-generic",
        "available_processors" : 8,
        "allocated_processors" : 8
      },
      "process" : {
        "refresh_interval_in_millis" : 1000,
        "id" : 21107,
        "mlockall" : false
      },
      "jvm" : {
        "pid" : 21107,
        "version" : "1.8.0_111",
        "vm_name" : "OpenJDK 64-Bit Server VM",
        "vm_version" : "25.111-b14",
        "vm_vendor" : "Oracle Corporation",
        "start_time_in_millis" : 1493125438981,
        "mem" : {
          "heap_init_in_bytes" : 2147483648,
          "heap_max_in_bytes" : 2077753344,
          "non_heap_init_in_bytes" : 2555904,
          "non_heap_max_in_bytes" : 0,
          "direct_max_in_bytes" : 2077753344
        },
        "gc_collectors" : [
          "ParNew",
          "ConcurrentMarkSweep"
        ],
        "memory_pools" : [...],
        "using_compressed_ordinary_object_pointers" : "true"
      },[...]
     },
      "plugins" : [
        {
          "name" : "x-pack",
          "version" : "5.3.0",
          "description" : "Elasticsearch Expanded Pack Plugin",
          "classname" : "org.elasticsearch.xpack.XPackPlugin"
        }
      ],
      "modules" : [
       [...]
        {
          "name" : "transport-netty3",
          "version" : "5.3.0",
          "description" : "Netty 3 based transport implementation",
          "classname" : "org.elasticsearch.transport.Netty3Plugin"
        },
        {
          "name" : "transport-netty4",
          "version" : "5.3.0",
          "description" : "Netty 4 based transport implementation",
          "classname" : "org.elasticsearch.transport.Netty4Plugin"
        }
      ],
      "ingest" : {
        "processors" : [
         [...]
        ]
      }
    }
  }
}

I also forgot to mention that I am working behind a proxy but it is configured to ignore localhost, 127.0.0.0/8 and ::1. Could it be interacting anyway ?

Any idea on how I could do further investigation ?

Thanks for your answers !

6

Problem solved. It was a hostname verification problem. In my elasticsearch.yml, I changed the _xpack.ssl.verification_mode option (default to full). First, I tried:

xpack.ssl.verification_mode: certificate

And it worked. The documentation is not really clear on this option, but I guess it just skips the hostname verification when enabling TLS.

Then, I put it back to full, and I set the network.host option (I guess it is the one used in the hostname verification ?)

xpack.ssl.verification_mode: full  # that's the default value
network.host: 127.0.0.1 # IP of my node

And then I could start elastic search with my single node, with the TLS node-to-node option enabled, without any errors. I also tried with 2 nodes and it had the same behavior. Handshake failed because of the hostname verification. But, when I set network.host "properly", everything worked fine.

I am still not sure about what is going on behind all this (why is TLS failing when I have only one node ? what "hostname" is compared to the certificate CN during the handshake ? And I'm a beginner with networks)... but the problem is solved.

Thanks for your help.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.