Failed to establish trust communication between es01, es02 and kibana

Elastic Stack Elasticsearch
,
5

Ok, I understand. Going back to the first configuration, given your edits to xpack.security.transport.ssl.certificate_authorities and new private key and certificate (with advanced SAN), I have such logs from es01:

{"type": "server", "timestamp": "2021-07-07T09:18:24,688Z", "level": "WARN", "component": "o.e.x.c.s.t.n.SecurityNetty4Transport", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/192.168.144.2:33070, remoteAddress=es02/192.168.144.3:9300, profile=default}" }
{"type": "server", "timestamp": "2021-07-07T09:18:25,368Z", "level": "WARN", "component": "o.e.c.s.DiagnosticTrustManager", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "failed to establish trust with client at [<unknown host>]; the client provided a certificate with subject name [1.2.840.113549.1.9.1=#161444416c656b68696e40616c666162616e6b2e7275,CN=elk-dkb-test.custome.domain.net,OU=custome,O=custome,L=custome,ST=custome,C=RU] and fingerprint [ff434cfdb315d1a23cb2302f2910889522b88616]; the certificate is issued by [CN=Sub2 CA 2012,DC=domain,DC=net] but the client did not provide a copy of the issuing certificate in the certificate chain; the issuing certificate with fingerprint [dd1a2abb0047a58642030c226fb73b8d0ed5ec96] is trusted in this ssl context ([xpack.security.transport.ssl])",
"stacktrace": ["sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS client authentication",
...

Based on this error:

sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS client authentication

and your recommendation from SSL between elastic nodes is failing I generated a new certificate with clientAuth and serverAuth extended key usage. Cluster got up:

Checking cluster health: https://192.168.160.4:9200/_cluster/health?pretty
{
  "cluster_name" : "es-docker-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

But I still get an error from kibana:

{"type":"log","@timestamp":"2021-07-07T14:58:56+00:00","tags":["fatal","root"],"pid":953,"message":"Error: ENOENT: no such file or directory, open '/usr/share/elasticsearch/config/certificates/Root_CA_2012.cer,/usr/share/elasticsearch/config/certificates/Sub2_CA_2012.cer'\n    at Object.openSync (fs.js:476:3)\n    at readFileSync (fs.js:377:35)\n    at readFile (/usr/share/kibana/src/core/server/elasticsearch/elasticsearch_config.js:378:31)\n    at readKeyAndCerts (/usr/share/kibana/src/core/server/elasticsearch/elasticsearch_config.js:362:21)\n    at new ElasticsearchConfig (/usr/share/kibana/src/core/server/elasticsearch/elasticsearch_config.js:297:9)\n    at MapSubscriber.project (/usr/share/kibana/src/core/server/elasticsearch/elasticsearch_service.js:49:108)\n    at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:49:35)\n    at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:55:26)\n    at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at DistinctUntilChangedSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/distinctUntilChanged.js:69:30)\n    at DistinctUntilChangedSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:55:26)\n    at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at ReplaySubject._subscribe (/usr/share/kibana/node_modules/rxjs/internal/ReplaySubject.js:80:28)\n    at ReplaySubject.Observable._trySubscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:44:25) {\n  errno: -2,\n  syscall: 'open',\n  code: 'ENOENT',\n  path: '/usr/share/elasticsearch/config/certificates/Root_CA_2012.cer,/usr/share/elasticsearch/config/certificates/Sub2_CA_2012.cer'\n}"}

 FATAL  Error: ENOENT: no such file or directory, open '/usr/share/elasticsearch/config/certificates/Root_CA_2012.cer,/usr/share/elasticsearch/config/certificates/Sub2_CA_2012.cer'

The excerpt of docker-compose.yml for kibana:

  kib01:
    image: docker.elastic.co/kibana/kibana:${VERSION}
    container_name: kib01
    depends_on: {"es01": {"condition": "service_healthy"}}
    ports:
      - 5601:5601
    environment:
      SERVERNAME: localhost
      ELASTICSEARCH_URL: https://es01:9200
      ELASTICSEARCH_HOSTS: https://es01:9200
      ELASTICSEARCH_USERNAME: kibana_system
      ELASTICSEARCH_PASSWORD: 4InJFrn0sog4oDH57n02
      SERVER_SSL_ENABLED: "true"
      ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: $CERTS_DIR/Root_CA_2012.cer,$CERTS_DIR/Sub2_CA_2012.cer
      SERVER_SSL_KEY: $CERTS_DIR/elk-dkb-test1_nopass.key
      SERVER_SSL_CERTIFICATE: $CERTS_DIR/elk-dkb-test1.cer
    volumes:
      - /home/dalekhin/certs:$CERTS_DIR
    networks:
      - elastic