Ok, I understand. Going back to the first configuration, given your edits to xpack.security.transport.ssl.certificate_authorities and new private key and certificate (with advanced SAN), I have such logs from es01:
{"type": "server", "timestamp": "2021-07-07T09:18:24,688Z", "level": "WARN", "component": "o.e.x.c.s.t.n.SecurityNetty4Transport", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/192.168.144.2:33070, remoteAddress=es02/192.168.144.3:9300, profile=default}" }
{"type": "server", "timestamp": "2021-07-07T09:18:25,368Z", "level": "WARN", "component": "o.e.c.s.DiagnosticTrustManager", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "failed to establish trust with client at [<unknown host>]; the client provided a certificate with subject name [1.2.840.113549.1.9.1=#161444416c656b68696e40616c666162616e6b2e7275,CN=elk-dkb-test.custome.domain.net,OU=custome,O=custome,L=custome,ST=custome,C=RU] and fingerprint [ff434cfdb315d1a23cb2302f2910889522b88616]; the certificate is issued by [CN=Sub2 CA 2012,DC=domain,DC=net] but the client did not provide a copy of the issuing certificate in the certificate chain; the issuing certificate with fingerprint [dd1a2abb0047a58642030c226fb73b8d0ed5ec96] is trusted in this ssl context ([xpack.security.transport.ssl])",
"stacktrace": ["sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS client authentication",
...
Based on this error:
sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS client authentication
and your recommendation from SSL between elastic nodes is failing I generated a new certificate with clientAuth and serverAuth extended key usage. Cluster got up:
Checking cluster health: https://192.168.160.4:9200/_cluster/health?pretty
{
"cluster_name" : "es-docker-cluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
But I still get an error from kibana:
{"type":"log","@timestamp":"2021-07-07T14:58:56+00:00","tags":["fatal","root"],"pid":953,"message":"Error: ENOENT: no such file or directory, open '/usr/share/elasticsearch/config/certificates/Root_CA_2012.cer,/usr/share/elasticsearch/config/certificates/Sub2_CA_2012.cer'\n at Object.openSync (fs.js:476:3)\n at readFileSync (fs.js:377:35)\n at readFile (/usr/share/kibana/src/core/server/elasticsearch/elasticsearch_config.js:378:31)\n at readKeyAndCerts (/usr/share/kibana/src/core/server/elasticsearch/elasticsearch_config.js:362:21)\n at new ElasticsearchConfig (/usr/share/kibana/src/core/server/elasticsearch/elasticsearch_config.js:297:9)\n at MapSubscriber.project (/usr/share/kibana/src/core/server/elasticsearch/elasticsearch_service.js:49:108)\n at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:49:35)\n at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:55:26)\n at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at DistinctUntilChangedSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/distinctUntilChanged.js:69:30)\n at DistinctUntilChangedSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:55:26)\n at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at ReplaySubject._subscribe (/usr/share/kibana/node_modules/rxjs/internal/ReplaySubject.js:80:28)\n at ReplaySubject.Observable._trySubscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:44:25) {\n errno: -2,\n syscall: 'open',\n code: 'ENOENT',\n path: '/usr/share/elasticsearch/config/certificates/Root_CA_2012.cer,/usr/share/elasticsearch/config/certificates/Sub2_CA_2012.cer'\n}"}
FATAL Error: ENOENT: no such file or directory, open '/usr/share/elasticsearch/config/certificates/Root_CA_2012.cer,/usr/share/elasticsearch/config/certificates/Sub2_CA_2012.cer'
The excerpt of docker-compose.yml for kibana:
kib01:
image: docker.elastic.co/kibana/kibana:${VERSION}
container_name: kib01
depends_on: {"es01": {"condition": "service_healthy"}}
ports:
- 5601:5601
environment:
SERVERNAME: localhost
ELASTICSEARCH_URL: https://es01:9200
ELASTICSEARCH_HOSTS: https://es01:9200
ELASTICSEARCH_USERNAME: kibana_system
ELASTICSEARCH_PASSWORD: 4InJFrn0sog4oDH57n02
SERVER_SSL_ENABLED: "true"
ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: $CERTS_DIR/Root_CA_2012.cer,$CERTS_DIR/Sub2_CA_2012.cer
SERVER_SSL_KEY: $CERTS_DIR/elk-dkb-test1_nopass.key
SERVER_SSL_CERTIFICATE: $CERTS_DIR/elk-dkb-test1.cer
volumes:
- /home/dalekhin/certs:$CERTS_DIR
networks:
- elastic