Failed to establish trust communication between es01, es02 and kibana

Elastic Stack Elasticsearch
,
1

Hello, I'm trying to configure TLS between es01, es02 and kibana (docker containers) with certificates from certificate chain [CA_cert - Intermediate_cert - Server_cert]. Below you can see the excerpt of the yml file where I'm setting up xpack security, and the error I am receiving when trying to ./bin/elasticsearch-setup-passwords auto from es01.

      - xpack.license.self_generated.type=trial
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.http.ssl.key=$CERTS_DIR/es01.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/Root_CA_2012.cer
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/Sub2_CA_2012.cer
      - xpack.security.http.ssl.certificate=$CERTS_DIR/es01.cer
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/Root_CA_2012.cer
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/Sub2_CA_2012.cer
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/es01.cer
      - xpack.security.transport.ssl.key=$CERTS_DIR/es01.key

Checking cluster health: https://192.168.48.3:9200/_cluster/health?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "master_not_discovered_exception",
        "reason" : null
      }
    ],
    "type" : "master_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}


Failed to determine the health of the cluster running at https://192.168.48.3:9200
Unexpected response code [503] from calling GET https://192.168.48.3:9200/_cluster/health?pretty
Cause: master_not_discovered_exception

It is recommended that you resolve the issues with your cluster before running elasticsearch-setup-passwords.
It is very likely that the password changes will fail when run against an unhealthy cluster.

It looks like the problem is that TLS is not being established between es01 and es02.

2021-07-02T09:54:04.219542828Z {"type": "server", "timestamp": "2021-07-02T09:54:04,218Z", "level": "WARN", "component": "o.e.c.s.DiagnosticTrustManager", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "failed to establish trust with client at [<unknown host>]; the client provided a certificate with subject name [CN=es02] and fingerprint [8ed4d2bd59aae371e46de7a224d83367f77f55c9]; the certificate is issued by [CN=Sub2 CA 2012,DC=mydomen,DC=net] but the client did not provide a copy of the issuing certificate in the certificate chain; the issuing certificate with fingerprint [dd1a2abb0047a58642030c226fb73b7d0ed5ec96] is trusted in this ssl context ([xpack.security.transport.ssl])",

Maybe I'm wrong. Can you help me?
P.S. it's my first case at this forum, sorry for formatting.

2

There should be an error and an exception stack trace in the logs as well.
This warning is helpful, but it's hard to make sense of it without the exact failure.

This situation is a bit weird.

Something (almost certainly node es02) is connecting to the transport port (9300) on node es01 with a certificate for CN=es02.
es01 doesn't not trust that certificate, even though it trusts certificates from that issuer (CN=Sub2 CA 2012,DC=mydomen,DC=net).
That implies that the problem with the trust is something different (like certificate validity dates, or certificate usage constraints). It will help if we can see more of the error messages.

I don't think this is the cause of your problem, but I'm pretty sure the config above isn't correct (but maybe it's a weird docker thing that I'm not aware of).
The 2nd environment variable definition will overwrite the first, so you will only configure Sub2 cert and not Root cert. I think you want

- xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/Root_CA_2012.cer,$CERTS_DIR/Sub2_CA_2012.cer
3

TimV, thanks for your answer. Exact failure is that cluster isn't health, isn't it?
I agree about the incorrectness of the config, but how should it look? I tried this

but got an error:

2021-07-06T06:59:07.843098542Z  FATAL  Error: ENOENT: no such file or directory, open '/usr/share/elasticsearch/config/certificates/Root_CA_2012.cer,/usr/share/elasticsearch/config/certificates/Sub2_CA_2012.cer'

At the same time, it is interesting that the container itself contains these certificates while the container is alive (kibana alived a few seconds and then got status "Exited"):

# docker exec -it kib01 /bin/bash
bash-4.4$ ls /usr/share/elasticsearch/config/certificates/
Root_CA_2012.cer  Sub2_CA_2012.cer  ...

I also tried config with PKCS12 (solutions 1 and 2 from Can't start ES 7.3.0 with x-pack security enabled):

      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.http.ssl.keystore.type=PKCS12
      - xpack.security.http.ssl.truststore.type=PKCS12
      - xpack.security.http.ssl.keystore.secure_password=
      - xpack.security.http.ssl.truststore.secure_password=
      - xpack.security.http.ssl.keystore.path=$CERTS_DIR/elk-dkb-test_nopas.p12
      - xpack.security.http.ssl.truststore.path=$CERTS_DIR/elk-dkb-test_nopas.p12
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.keystore.type=PKCS12
      - xpack.security.transport.ssl.truststore.type=PKCS12
      - xpack.security.transport.ssl.keystore.path=$CERTS_DIR/elk-dkb-test_nopas.p12
      - xpack.security.transport.ssl.truststore.path=$CERTS_DIR/elk-dkb-test_nopas.p12

but received an errors, solution 1:

2021-07-06T09:31:52.230550017Z "stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager]; nested: IOException[keystore password was incorrect]; nested: UnrecoverableKeyException[failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.];",
...
2021-07-06T09:31:52.230663591Z "Caused by: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl]",
...
2021-07-06T09:31:52.230850972Z "Caused by: org.elasticsearch.ElasticsearchException: failed to initialize SSL TrustManager",

solution 2:

      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.http.ssl.keystore.type=PKCS12
      - xpack.security.http.ssl.truststore.type=PKCS12
      - xpack.security.http.ssl.keystore.secure_password=$SECPAS
      - xpack.security.http.ssl.truststore.secure_password=$SECPAS
      - xpack.security.http.ssl.keystore.path=$CERTS_DIR/elk-dkb-test.p12
      - xpack.security.http.ssl.truststore.path=$CERTS_DIR/elk-dkb-test.p12
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.keystore.type=PKCS12
      - xpack.security.transport.ssl.truststore.type=PKCS12
      - xpack.security.transport.ssl.keystore.secure_password=$SECPAS
      - xpack.security.transport.ssl.truststore.secure_password=$SECPAS
      - xpack.security.transport.ssl.keystore.path=$CERTS_DIR/elk-dkb-test.p12
      - xpack.security.transport.ssl.truststore.path=$CERTS_DIR/elk-dkb-test.p12



2021-07-06T12:02:03.410147079Z {"type": "server", "timestamp": "2021-07-06T12:02:03,404Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "uncaught exception in thread [main]",
2021-07-06T12:02:03.410179378Z "stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Setting [xpack.security.http.ssl.truststore.secure_password] is a secure setting and must be stored inside the Elasticsearch keystore, but was found inside elasticsearch.yml",
...
4

No, there should be a precise failure message for the failure to establish TLS.

What you have quoted is a diagnostic warning, which is helpful, but it isn't the actual error. The real error will be something like SSLHandshakeException: <<potentially important details here>>

5

Ok, I understand. Going back to the first configuration, given your edits to xpack.security.transport.ssl.certificate_authorities and new private key and certificate (with advanced SAN), I have such logs from es01:

{"type": "server", "timestamp": "2021-07-07T09:18:24,688Z", "level": "WARN", "component": "o.e.x.c.s.t.n.SecurityNetty4Transport", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/192.168.144.2:33070, remoteAddress=es02/192.168.144.3:9300, profile=default}" }
{"type": "server", "timestamp": "2021-07-07T09:18:25,368Z", "level": "WARN", "component": "o.e.c.s.DiagnosticTrustManager", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "failed to establish trust with client at [<unknown host>]; the client provided a certificate with subject name [1.2.840.113549.1.9.1=#161444416c656b68696e40616c666162616e6b2e7275,CN=elk-dkb-test.custome.domain.net,OU=custome,O=custome,L=custome,ST=custome,C=RU] and fingerprint [ff434cfdb315d1a23cb2302f2910889522b88616]; the certificate is issued by [CN=Sub2 CA 2012,DC=domain,DC=net] but the client did not provide a copy of the issuing certificate in the certificate chain; the issuing certificate with fingerprint [dd1a2abb0047a58642030c226fb73b8d0ed5ec96] is trusted in this ssl context ([xpack.security.transport.ssl])",
"stacktrace": ["sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS client authentication",
...

Based on this error:

sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS client authentication

and your recommendation from SSL between elastic nodes is failing I generated a new certificate with clientAuth and serverAuth extended key usage. Cluster got up:

Checking cluster health: https://192.168.160.4:9200/_cluster/health?pretty
{
  "cluster_name" : "es-docker-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

But I still get an error from kibana:

{"type":"log","@timestamp":"2021-07-07T14:58:56+00:00","tags":["fatal","root"],"pid":953,"message":"Error: ENOENT: no such file or directory, open '/usr/share/elasticsearch/config/certificates/Root_CA_2012.cer,/usr/share/elasticsearch/config/certificates/Sub2_CA_2012.cer'\n    at Object.openSync (fs.js:476:3)\n    at readFileSync (fs.js:377:35)\n    at readFile (/usr/share/kibana/src/core/server/elasticsearch/elasticsearch_config.js:378:31)\n    at readKeyAndCerts (/usr/share/kibana/src/core/server/elasticsearch/elasticsearch_config.js:362:21)\n    at new ElasticsearchConfig (/usr/share/kibana/src/core/server/elasticsearch/elasticsearch_config.js:297:9)\n    at MapSubscriber.project (/usr/share/kibana/src/core/server/elasticsearch/elasticsearch_service.js:49:108)\n    at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:49:35)\n    at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:55:26)\n    at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at DistinctUntilChangedSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/distinctUntilChanged.js:69:30)\n    at DistinctUntilChangedSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:55:26)\n    at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at ReplaySubject._subscribe (/usr/share/kibana/node_modules/rxjs/internal/ReplaySubject.js:80:28)\n    at ReplaySubject.Observable._trySubscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:44:25) {\n  errno: -2,\n  syscall: 'open',\n  code: 'ENOENT',\n  path: '/usr/share/elasticsearch/config/certificates/Root_CA_2012.cer,/usr/share/elasticsearch/config/certificates/Sub2_CA_2012.cer'\n}"}

 FATAL  Error: ENOENT: no such file or directory, open '/usr/share/elasticsearch/config/certificates/Root_CA_2012.cer,/usr/share/elasticsearch/config/certificates/Sub2_CA_2012.cer'

The excerpt of docker-compose.yml for kibana:

  kib01:
    image: docker.elastic.co/kibana/kibana:${VERSION}
    container_name: kib01
    depends_on: {"es01": {"condition": "service_healthy"}}
    ports:
      - 5601:5601
    environment:
      SERVERNAME: localhost
      ELASTICSEARCH_URL: https://es01:9200
      ELASTICSEARCH_HOSTS: https://es01:9200
      ELASTICSEARCH_USERNAME: kibana_system
      ELASTICSEARCH_PASSWORD: 4InJFrn0sog4oDH57n02
      SERVER_SSL_ENABLED: "true"
      ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: $CERTS_DIR/Root_CA_2012.cer,$CERTS_DIR/Sub2_CA_2012.cer
      SERVER_SSL_KEY: $CERTS_DIR/elk-dkb-test1_nopass.key
      SERVER_SSL_CERTIFICATE: $CERTS_DIR/elk-dkb-test1.cer
    volumes:
      - /home/dalekhin/certs:$CERTS_DIR
    networks:
      - elastic
6

Sorry, I'm not sure how to pass multiple CA certificates to Kibana in Docker.
You may need to ask in the Kibana forum.

I don't think you need both - you should be able to anchor trust on just the intermediate (Sub2_CA_2012) certificate.

7

Thank you very match for help, Tim!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

9

It works by putting the certificates configured for ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: into a chained certificate file. Simply open the $CERTS_DIR/Root_CA_2012.cer and $CERTS_DIR/Sub2_CA_2012.cer files with a text editor and copy their content into a new file. The rootCA needs to be at the bottom and the intermediates need to go above it in the order of their chain. Then configure this new files (the certificate chain) for ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES:.

See OpenSSL - User - check certificate chain in a pem file for examples.

2 Likes