After migrating from a 7.8.1 stack to 7.16.3, Kibana will intermittently restart, with the last log being:
TypeError: Cannot read properties of undefined (reading 'isServer')
at TLSWrap.onerror (node:_tls_wrap:411:27)
Upgrading to 7.17.1, this issue still persisted.
We are using a docker swarm cluster with 3 GCP Confidential VMs. There is an Elasticsearch service per instance, and we have tried 1-3 kibana replicas, with all replicas still failing intermittently.
We had restored the .kibana_1 and .security7 index from the old cluster when migrating, and then upgraded the stack.
In Elasticsearch we get this error at a similar time to the kibana one.
... caught exception while handling client http traffic, closing connection ...
... javax.net.ssl.SSLHandshakeException: Received fatal alert: protocol_version
...
We created Elasticsearch certificates using the elasticsearch-certutil tool. The environment variables we use for the kibana service are:
SERVER_NAME: server-name
SERVER_PUBLICBASEURL: url
ELASTICSEARCH_HOSTS: '["https://elasticsearch1:9200","https://elasticsearch2:9200","https://elasticsearch3:9200"]'
SERVER_HOST: 0.0.0.0
SERVER_SSL_ENABLED: "true"
SERVER_SSL_CERTIFICATE: *server_cert
SERVER_SSL_KEY: *server_key
XPACK_SECURITY_SESSION_IDLETIMEOUT: "1h"
XPACK_SECURITY_SESSION_LIFESPAN: "30d"
ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: *certificate_authorities
ELASTICSEARCH_SSL_VERIFICATIONMODE: certificate
ELASTICSEARCH_USERNAME: $ELASTIC_USERNAME
ELASTICSEARCH_PASSWORD: $ELASTIC_PASSWORD
XPACK_SECURITY_ENCRYPTIONKEY: $XPACK_SECURITY_ENCRYPTIONKEY
XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY: $XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY
XPACK_REPORTING_ENCRYPTIONKEY: $XPACK_REPORTING_ENCRYPTIONKEY
The environment variables for the Elasticsearch service are:
ES_JAVA_OPTS: "-Xms2g -Xmx2g"
ELASTIC_PASSWORD: $ELASTIC_PASSWORD
GCP_CREDENTIALS: *gcp_credentials
GCP_BUCKET: $GCP_BUCKET
GCP_BUCKET_FOLDER: $GCP_BUCKET_FOLDER
network.publish_host: _eth0_
network.bind_host: _eth0_,_eth1_
discovery.seed_hosts: elasticsearch1,elasticsearch2,elasticsearch3
cluster.initial_master_nodes: elasticsearch1,elasticsearch2,elasticsearch3
xpack.security.enabled: "true"
xpack.security.ssl.diagnose.trust: "true"
xpack.security.http.ssl.enabled: "true"
xpack.security.http.ssl.key: *elasticsearch_key
xpack.security.http.ssl.certificate_authorities: *certificate_authorities
xpack.security.http.ssl.certificate: *elasticsearch_certificate
xpack.security.http.ssl.verification_mode: certificate
xpack.security.transport.ssl.enabled: "true"
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.certificate_authorities: *certificate_authorities
xpack.security.transport.ssl.certificate: *elasticsearch_certificate
xpack.security.transport.ssl.key: *elasticsearch_key
This was not happening on version 7.8.1 and we cannot replicate this issue in a dev environment so are stumped on how to fix it. We tried regenerating certs, but no luck.
Any help would be appreciated, thank you