Kibana 17.1 restarting after: "TypeError: Cannot read properties of undefined (reading 'isServer')"

Elastic Stack Kibana
1

After migrating from a 7.8.1 stack to 7.16.3, Kibana will intermittently restart, with the last log being:

TypeError: Cannot read properties of undefined (reading 'isServer')
    at TLSWrap.onerror (node:_tls_wrap:411:27)

Upgrading to 7.17.1, this issue still persisted.
We are using a docker swarm cluster with 3 GCP Confidential VMs. There is an Elasticsearch service per instance, and we have tried 1-3 kibana replicas, with all replicas still failing intermittently.

We had restored the .kibana_1 and .security7 index from the old cluster when migrating, and then upgraded the stack.

In Elasticsearch we get this error at a similar time to the kibana one.

... caught exception while handling client http traffic, closing connection ...
... javax.net.ssl.SSLHandshakeException: Received fatal alert: protocol_version
...

We created Elasticsearch certificates using the elasticsearch-certutil tool. The environment variables we use for the kibana service are:

SERVER_NAME: server-name
SERVER_PUBLICBASEURL: url
ELASTICSEARCH_HOSTS: '["https://elasticsearch1:9200","https://elasticsearch2:9200","https://elasticsearch3:9200"]' 
SERVER_HOST: 0.0.0.0
SERVER_SSL_ENABLED: "true"
SERVER_SSL_CERTIFICATE: *server_cert
SERVER_SSL_KEY: *server_key
XPACK_SECURITY_SESSION_IDLETIMEOUT: "1h"
XPACK_SECURITY_SESSION_LIFESPAN: "30d"
ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: *certificate_authorities
ELASTICSEARCH_SSL_VERIFICATIONMODE: certificate
ELASTICSEARCH_USERNAME: $ELASTIC_USERNAME
ELASTICSEARCH_PASSWORD: $ELASTIC_PASSWORD
XPACK_SECURITY_ENCRYPTIONKEY: $XPACK_SECURITY_ENCRYPTIONKEY
XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY: $XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY
XPACK_REPORTING_ENCRYPTIONKEY: $XPACK_REPORTING_ENCRYPTIONKEY

The environment variables for the Elasticsearch service are:

ES_JAVA_OPTS: "-Xms2g -Xmx2g"
ELASTIC_PASSWORD: $ELASTIC_PASSWORD
GCP_CREDENTIALS: *gcp_credentials
GCP_BUCKET: $GCP_BUCKET
GCP_BUCKET_FOLDER: $GCP_BUCKET_FOLDER
network.publish_host: _eth0_
network.bind_host: _eth0_,_eth1_ 
discovery.seed_hosts: elasticsearch1,elasticsearch2,elasticsearch3
cluster.initial_master_nodes: elasticsearch1,elasticsearch2,elasticsearch3
xpack.security.enabled: "true"
xpack.security.ssl.diagnose.trust: "true"
xpack.security.http.ssl.enabled: "true"
xpack.security.http.ssl.key: *elasticsearch_key
xpack.security.http.ssl.certificate_authorities: *certificate_authorities
xpack.security.http.ssl.certificate: *elasticsearch_certificate
xpack.security.http.ssl.verification_mode: certificate
xpack.security.transport.ssl.enabled: "true"
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.certificate_authorities: *certificate_authorities
xpack.security.transport.ssl.certificate: *elasticsearch_certificate
xpack.security.transport.ssl.key: *elasticsearch_key

This was not happening on version 7.8.1 and we cannot replicate this issue in a dev environment so are stumped on how to fix it. We tried regenerating certs, but no luck.

Any help would be appreciated, thank you

2

we cannot replicate this issue in a dev environment

Hmm, do you have the same JDK version in all ES containers? Do you have the same JDK versions in dev and prod?

3

We are using the same official images for all containers: docker.elastic.co/elasticsearch/elasticsearch:7.17.1.
So ES containers would have the same JDK version, which seems to be 17.0.2

4

Something that we have found is that if there are saved objects in the .kibana index the issue is far more likely to happen. Particularly if there are a lot. For example, if downloading in the default security rules, then Kibana starts to have this problem and can even become unusable because it ends up in a restart loop

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.