Waiting for the transport certificates

SeibertronSS · April 12, 2023, 2:33am

Hi, I followed ECK's documentation to install elasticsearch using Elastic Operator. My elasticsearch pod is stuck in Init state when I use the provided quick start, here is a snippet of its log

Starting init script
Linking /mnt/elastic-internal/xpack-file-realm/users to /usr/share/elasticsearch/config/users
Linking /mnt/elastic-internal/xpack-file-realm/roles.yml to /usr/share/elasticsearch/config/roles.yml
Linking /mnt/elastic-internal/xpack-file-realm/users_roles to /usr/share/elasticsearch/config/users_roles
Linking /mnt/elastic-internal/elasticsearch-config/elasticsearch.yml to /usr/share/elasticsearch/config/elasticsearch.yml
Linking /mnt/elastic-internal/unicast-hosts/unicast_hosts.txt to /usr/share/elasticsearch/config/unicast_hosts.txt
Linking /mnt/elastic-internal/xpack-file-realm/service_tokens to /usr/share/elasticsearch/config/service_tokens
File linking duration: 0 sec.
Copying /usr/share/elasticsearch/config/* to /mnt/elastic-internal/elasticsearch-config-local/
'/usr/share/elasticsearch/config/elasticsearch-plugins.example.yml' -> '/mnt/elastic-internal/elasticsearch-config-local/elasticsearch-plugins.example.yml'
'/usr/share/elasticsearch/config/elasticsearch.yml' -> '/mnt/elastic-internal/elasticsearch-config-local/elasticsearch.yml'
'/usr/share/elasticsearch/config/http-certs' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs'
'/usr/share/elasticsearch/config/http-certs/..2023_04_12_02_00_28.487846342' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/..2023_04_12_02_00_28.487846342'
'/usr/share/elasticsearch/config/http-certs/..2023_04_12_02_00_28.487846342/tls.key' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/..2023_04_12_02_00_28.487846342/tls.key'
'/usr/share/elasticsearch/config/http-certs/..2023_04_12_02_00_28.487846342/ca.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/..2023_04_12_02_00_28.487846342/ca.crt'
'/usr/share/elasticsearch/config/http-certs/..2023_04_12_02_00_28.487846342/tls.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/..2023_04_12_02_00_28.487846342/tls.crt'
'/usr/share/elasticsearch/config/http-certs/tls.key' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/tls.key'
'/usr/share/elasticsearch/config/http-certs/ca.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/ca.crt'
'/usr/share/elasticsearch/config/http-certs/tls.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/tls.crt'
'/usr/share/elasticsearch/config/http-certs/..data' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/..data'
'/usr/share/elasticsearch/config/jvm.options' -> '/mnt/elastic-internal/elasticsearch-config-local/jvm.options'
'/usr/share/elasticsearch/config/jvm.options.d' -> '/mnt/elastic-internal/elasticsearch-config-local/jvm.options.d'
'/usr/share/elasticsearch/config/log4j2.file.properties' -> '/mnt/elastic-internal/elasticsearch-config-local/log4j2.file.properties'
'/usr/share/elasticsearch/config/log4j2.properties' -> '/mnt/elastic-internal/elasticsearch-config-local/log4j2.properties'
'/usr/share/elasticsearch/config/operator' -> '/mnt/elastic-internal/elasticsearch-config-local/operator'
'/usr/share/elasticsearch/config/operator/..2023_04_12_02_00_28.194504000' -> '/mnt/elastic-internal/elasticsearch-config-local/operator/..2023_04_12_02_00_28.194504000'
'/usr/share/elasticsearch/config/operator/..2023_04_12_02_00_28.194504000/settings.json' -> '/mnt/elastic-internal/elasticsearch-config-local/operator/..2023_04_12_02_00_28.194504000/settings.json'
'/usr/share/elasticsearch/config/operator/settings.json' -> '/mnt/elastic-internal/elasticsearch-config-local/operator/settings.json'
'/usr/share/elasticsearch/config/operator/..data' -> '/mnt/elastic-internal/elasticsearch-config-local/operator/..data'
'/usr/share/elasticsearch/config/role_mapping.yml' -> '/mnt/elastic-internal/elasticsearch-config-local/role_mapping.yml'
'/usr/share/elasticsearch/config/roles.yml' -> '/mnt/elastic-internal/elasticsearch-config-local/roles.yml'
'/usr/share/elasticsearch/config/service_tokens' -> '/mnt/elastic-internal/elasticsearch-config-local/service_tokens'
'/usr/share/elasticsearch/config/transport-remote-certs' -> '/mnt/elastic-internal/elasticsearch-config-local/transport-remote-certs'
'/usr/share/elasticsearch/config/transport-remote-certs/..2023_04_12_02_00_28.794736242' -> '/mnt/elastic-internal/elasticsearch-config-local/transport-remote-certs/..2023_04_12_02_00_28.794736242'
'/usr/share/elasticsearch/config/transport-remote-certs/..2023_04_12_02_00_28.794736242/ca.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/transport-remote-certs/..2023_04_12_02_00_28.794736242/ca.crt'
'/usr/share/elasticsearch/config/transport-remote-certs/ca.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/transport-remote-certs/ca.crt'
'/usr/share/elasticsearch/config/transport-remote-certs/..data' -> '/mnt/elastic-internal/elasticsearch-config-local/transport-remote-certs/..data'
'/usr/share/elasticsearch/config/unicast_hosts.txt' -> '/mnt/elastic-internal/elasticsearch-config-local/unicast_hosts.txt'
'/usr/share/elasticsearch/config/users' -> '/mnt/elastic-internal/elasticsearch-config-local/users'
'/usr/share/elasticsearch/config/users_roles' -> '/mnt/elastic-internal/elasticsearch-config-local/users_roles'
Empty dir /usr/share/elasticsearch/plugins
Copying /usr/share/elasticsearch/bin/* to /mnt/elastic-internal/elasticsearch-bin-local/
'/usr/share/elasticsearch/bin/elasticsearch' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch'
'/usr/share/elasticsearch/bin/elasticsearch-certgen' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-certgen'
'/usr/share/elasticsearch/bin/elasticsearch-certutil' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-certutil'
'/usr/share/elasticsearch/bin/elasticsearch-cli' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-cli'
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-create-enrollment-token'
'/usr/share/elasticsearch/bin/elasticsearch-croneval' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-croneval'
'/usr/share/elasticsearch/bin/elasticsearch-env' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-env'
'/usr/share/elasticsearch/bin/elasticsearch-env-from-file' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-env-from-file'
'/usr/share/elasticsearch/bin/elasticsearch-geoip' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-geoip'
'/usr/share/elasticsearch/bin/elasticsearch-keystore' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-keystore'
'/usr/share/elasticsearch/bin/elasticsearch-node' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-node'
'/usr/share/elasticsearch/bin/elasticsearch-plugin' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-plugin'
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-reconfigure-node'
'/usr/share/elasticsearch/bin/elasticsearch-reset-password' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-reset-password'
'/usr/share/elasticsearch/bin/elasticsearch-saml-metadata' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-saml-metadata'
'/usr/share/elasticsearch/bin/elasticsearch-service-tokens' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-service-tokens'
'/usr/share/elasticsearch/bin/elasticsearch-setup-passwords' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-setup-passwords'
'/usr/share/elasticsearch/bin/elasticsearch-shard' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-shard'
'/usr/share/elasticsearch/bin/elasticsearch-sql-cli' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-sql-cli'
'/usr/share/elasticsearch/bin/elasticsearch-sql-cli-8.7.0.jar' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-sql-cli-8.7.0.jar'
'/usr/share/elasticsearch/bin/elasticsearch-syskeygen' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-syskeygen'
'/usr/share/elasticsearch/bin/elasticsearch-users' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-users'
Files copy duration: 0 sec.
chown duration: 0 sec.
waiting for the transport certificates (/mnt/elastic-internal/transport-certificates/quickstart-es-default-0.tls.key)

I found that elasticsearch provides the required files to the pod through a secret, but there is only one ca.crt file under the path /usr/share/elasticsearch/config/transport-remote-certs in the pod.

My kubernetes version is 1.19. When I used an experimental Pod to mount the same secret, I found that I could see all the files.

michael.morello · April 12, 2023, 4:54am

Hi,

The operator generates the transport certificates. Checking if there is a problem in its logs can help. The situation you describe can be explained by the fact that the operator is not running, or the Pod has no IP (kubectl get pods -o wide) for example.

SeibertronSS · April 12, 2023, 5:21am

The pod of the operator is Running

NAME                 READY   STATUS    RESTARTS   AGE     IP             NODE          NOMINATED NODE   READINESS GATES
elastic-operator-0   1/1     Running   0          2m19s   11.xx.12.xxx   11.xx.1.xxx   <none>           <none>

system · May 10, 2023, 5:22am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.