It is not the first time that I have faced this error, in an operational installation, that after a restart of either the server or a restart of the elasticcsearch + kibana service, has as a result this fatal message that makes kibana unusable. Elasticsearch is still working perfectly.
I have seen more than 50 articles on the subject. Something that seems quite recurring, so I don't expect much help or solution.
But my data goes here.
Elastic and Kibana: 8.2.2
Installed via official repo of Ubuntu 22.04
Installed with defaults (https, cert, etc.)
What was he doing
- It occurred to me (just like on another occasion) to start kibana, when Elasticsearch was not yet active.
- I went back, restarting Elasticsearch and kibana and allowing a 30 second window for the only node (it's a local development install) to give it enough time to go yellow.
Logs
Last restart of kibana
jun 10 07:44:38 abkrim-nox kibana[5713]: [2022-06-10T07:44:38.833+02:00][WARN ][plugins-system.standard] "eventLog" plugin didn't stop in 30sec., move on to the next.
jun 10 07:44:38 abkrim-nox kibana[5713]: [2022-06-10T07:44:38.842+02:00][FATAL][root] no elements in sequence
jun 10 07:44:38 abkrim-nox kibana[5713]: FATAL EmptyError: no elements in sequence
jun 10 07:44:38 abkrim-nox systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
jun 10 07:44:38 abkrim-nox systemd[1]: kibana.service: Failed with result 'exit-code'.
jun 10 07:44:38 abkrim-nox systemd[1]: Stopped Kibana.
jun 10 07:44:38 abkrim-nox systemd[1]: kibana.service: Consumed 19.600s CPU time.
jun 10 07:44:38 abkrim-nox systemd[1]: Started Kibana.
jun 10 07:44:49 abkrim-nox kibana[11844]: [2022-06-10T07:44:49.066+02:00][INFO ][plugins-service] Plugin "cloudSecurityPosture" is disabled.
jun 10 07:44:49 abkrim-nox kibana[11844]: [2022-06-10T07:44:49.083+02:00][INFO ][plugins-service] Plugin "metricsEntities" is disabled.
jun 10 07:44:49 abkrim-nox kibana[11844]: [2022-06-10T07:44:49.147+02:00][INFO ][http.server.Preboot] http server running at http://localhost:5601
jun 10 07:44:49 abkrim-nox kibana[11844]: [2022-06-10T07:44:49.188+02:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
jun 10 07:44:49 abkrim-nox kibana[11844]: [2022-06-10T07:44:49.236+02:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this clust>
jun 10 07:44:49 abkrim-nox kibana[11844]: [2022-06-10T07:44:49.404+02:00][INFO ][plugins-system.standard] Setting up [117] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licen>
jun 10 07:44:49 abkrim-nox kibana[11844]: [2022-06-10T07:44:49.417+02:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 74490b16-281d-4842-a6d1-e3d3efb052b7
jun 10 07:44:49 abkrim-nox kibana[11844]: [2022-06-10T07:44:49.532+02:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpac>
jun 10 07:44:49 abkrim-nox kibana[11844]: [2022-06-10T07:44:49.532+02:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
jun 10 07:44:49 abkrim-nox kibana[11844]: [2022-06-10T07:44:49.548+02:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpac>
jun 10 07:44:49 abkrim-nox kibana[11844]: [2022-06-10T07:44:49.548+02:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
jun 10 07:44:49 abkrim-nox kibana[11844]: [2022-06-10T07:44:49.563+02:00][WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xp>
jun 10 07:44:49 abkrim-nox kibana[11844]: [2022-06-10T07:44:49.571+02:00][WARN ][plugins.encryptedSavedObjects] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObj>
jun 10 07:44:49 abkrim-nox kibana[11844]: [2022-06-10T07:44:49.585+02:00][WARN ][plugins.actions] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptio>
jun 10 07:44:49 abkrim-nox kibana[11844]: [2022-06-10T07:44:49.605+02:00][WARN ][plugins.alerting] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encrypti>
jun 10 07:44:49 abkrim-nox kibana[11844]: [2022-06-10T07:44:49.617+02:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
jun 10 07:44:50 abkrim-nox kibana[11844]: [2022-06-10T07:44:50.116+02:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 22.04 OS. Automatically ena>
jun 10 07:44:50 abkrim-nox kibana[11844]: [2022-06-10T07:44:50.614+02:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
jun 10 07:46:50 abkrim-nox kibana[11844]: [2022-06-10T07:46:50.150+02:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. Request timed out
Steeps for correct problem
Well, looking at the logs it seems that during the installation process something didn't tell me about the need to use the encryptionKey, at least in my installation that followed the steps carefully.
More strange is that for 10 days with several restarts it has worked and has been working with Kibana to test it and to work with the indexes that I have been creating, but this time it stops
Okay, I'm going to it. kibana-encryption-keys
sudo /usr/share/kibana/bin/kibana-encryption-keys generate -i
## Kibana Encryption Key Generation Utility
The 'generate' command guides you through the process of setting encryption keys for:
xpack.encryptedSavedObjects.encryptionKey
Used to encrypt stored objects such as dashboards and visualizations
https://www.elastic.co/guide/en/kibana/current/xpack-security-secure-saved-objects.html#xpack-security-secure-saved-objects
xpack.reporting.encryptionKey
Used to encrypt saved reports
https://www.elastic.co/guide/en/kibana/current/reporting-settings-kb.html#general-reporting-settings
xpack.security.encryptionKey
Used to encrypt session information
https://www.elastic.co/guide/en/kibana/current/security-settings-kb.html#security-session-and-cookie-settings
Already defined settings are ignored and can be regenerated using the --force flag. Check the documentation links for instructions on how to rotate encryption keys.
Definitions should be set in the kibana.yml used configure Kibana.
This tool will ask you a number of questions in order to generate the right set of keys for your needs.
Set xpack.encryptedSavedObjects.encryptionKey? [y/N] y
Set xpack.reporting.encryptionKey? [y/N] y
Set xpack.security.encryptionKey? [y/N] y
The following keys were generated:
xpack.encryptedSavedObjects.encryptionKey
xpack.reporting.encryptionKey
xpack.security.encryptionKey
Save generated keys to a sample Kibana configuration file? [y/N] y
What filename should be used for the sample Kibana config file? [/etc/kibana/kibana.sample.yml]):
Wrote configuration to /etc/kibana/kibana.sample.yml
After read file /etc/kibana/kibana.sample.yml and add xpack.* pairs into kibana.yml
un 10 08:03:06 abkrim-nox systemd[1]: Started Kibana.
jun 10 08:03:15 abkrim-nox kibana[14303]: [2022-06-10T08:03:15.510+02:00][INFO ][plugins-service] Plugin "cloudSecurityPosture" is disabled.
jun 10 08:03:15 abkrim-nox kibana[14303]: [2022-06-10T08:03:15.524+02:00][INFO ][plugins-service] Plugin "metricsEntities" is disabled.
jun 10 08:03:15 abkrim-nox kibana[14303]: [2022-06-10T08:03:15.593+02:00][INFO ][http.server.Preboot] http server running at http://localhost:5601
jun 10 08:03:15 abkrim-nox kibana[14303]: [2022-06-10T08:03:15.631+02:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
jun 10 08:03:15 abkrim-nox kibana[14303]: [2022-06-10T08:03:15.678+02:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this clust>
jun 10 08:03:15 abkrim-nox kibana[14303]: [2022-06-10T08:03:15.847+02:00][INFO ][plugins-system.standard] Setting up [117] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licen>
jun 10 08:03:15 abkrim-nox kibana[14303]: [2022-06-10T08:03:15.859+02:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 74490b16-281d-4842-a6d1-e3d3efb052b7
jun 10 08:03:15 abkrim-nox kibana[14303]: [2022-06-10T08:03:15.969+02:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
jun 10 08:03:15 abkrim-nox kibana[14303]: [2022-06-10T08:03:15.985+02:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
jun 10 08:03:16 abkrim-nox kibana[14303]: [2022-06-10T08:03:16.046+02:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
jun 10 08:03:16 abkrim-nox kibana[14303]: [2022-06-10T08:03:16.520+02:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 22.04 OS. Automatically ena>
jun 10 08:03:16 abkrim-nox kibana[14303]: [2022-06-10T08:03:16.891+02:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
There are no more error messages in the journal, worse still in the log
un 10 08:03:06 abkrim-nox systemd[1]: Started Kibana.
jun 10 08:03:15 abkrim-nox kibana[14303]: [2022-06-10T08:03:15.510+02:00][INFO ][plugins-service] Plugin "cloudSecurityPosture" is disabled.
jun 10 08:03:15 abkrim-nox kibana[14303]: [2022-06-10T08:03:15.524+02:00][INFO ][plugins-service] Plugin "metricsEntities" is disabled.
jun 10 08:03:15 abkrim-nox kibana[14303]: [2022-06-10T08:03:15.593+02:00][INFO ][http.server.Preboot] http server running at http://localhost:5601
jun 10 08:03:15 abkrim-nox kibana[14303]: [2022-06-10T08:03:15.631+02:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
jun 10 08:03:15 abkrim-nox kibana[14303]: [2022-06-10T08:03:15.678+02:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this clust>
jun 10 08:03:15 abkrim-nox kibana[14303]: [2022-06-10T08:03:15.847+02:00][INFO ][plugins-system.standard] Setting up [117] plugins: [translations,monitoringCollection,licensing,globalSearch,globalSearchProviders,features,mapsEms,licen>
jun 10 08:03:15 abkrim-nox kibana[14303]: [2022-06-10T08:03:15.859+02:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 74490b16-281d-4842-a6d1-e3d3efb052b7
jun 10 08:03:15 abkrim-nox kibana[14303]: [2022-06-10T08:03:15.969+02:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
jun 10 08:03:15 abkrim-nox kibana[14303]: [2022-06-10T08:03:15.985+02:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
jun 10 08:03:16 abkrim-nox kibana[14303]: [2022-06-10T08:03:16.046+02:00][INFO ][plugins.ruleRegistry] Installing common resources shared between all indices
jun 10 08:03:16 abkrim-nox kibana[14303]: [2022-06-10T08:03:16.520+02:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 22.04 OS. Automatically ena>
jun 10 08:03:16 abkrim-nox kibana[14303]: [2022-06-10T08:03:16.891+02:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/x-pack/plugins/screenshotting/chromium/headless_shell-linux_x64/headless_shell
Log
{"ecs":{"version":"8.0.0"},"@timestamp":"2022-06-10T08:05:16.558+02:00","message":"Unable to retrieve version information from Elasticsearch nodes.Request timed out","log":{"level":"ERROR","logger":"elasticsearch-service"},"process":{"pid":14303},"trace":{"id ":"f753f98f13918bb9f471c5c36b9b7a10"},"transaction":{"id":"92bb002108ce8be5"}}
But if I curl my node...
curl --cacert /home/abkrim/http_ca.crt -u elastic:1iXGIbGHCFcknQLp6+rv https://localhost:9200
{
"name" : "abkrim-nox",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "E_d31aTxSaKlUIIQhOKZkw",
"version" : {
"number" : "8.2.2",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "9876968ef3c745186b94fdabd4483e01499224ef",
"build_date" : "2022-05-25T15:47:06.259735307Z",
"build_snapshot" : false,
"lucene_version" : "9.1.0",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
Certainly this situation despairs me. I can continue working with Elasticsearch, but kibana left me unemployed.
Grateful if I get any ideas or help.
My config
elastic
sudo grep -Ev '^#|^$' /etc/elasticsearch/elasticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["abkrim-nox"]
http.host: 0.0.0.0
kibana
sudo grep -Ev '^#|^$' /etc/kibana/kibana.yml
server.host: "localhost"
logging:
appenders:
file:
type: file
fileName: /var/log/kibana/kibana.log
layout:
type: json
root:
appenders:
- default
- file
pid.file: /run/kibana/kibana.pid
elasticsearch.hosts: ['https://10.8.0.2:9200']
elasticsearch.serviceAccountToken: AAEAAWVsYXN0aWMva2liYW5hL2Vucm9sbC1wcm9jZXNzLXRva2VuLTE2NTMwNzUzMDM3NjA6ejNxRkVYOTlUbk9Qa0NobE1qdFBYQQ
elasticsearch.ssl.certificateAuthorities: [/var/lib/kibana/ca_1653075304659.crt]
xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: ['https://10.8.0.2:9200'], ca_trusted_fingerprint: 6f4c6725d31eda8cb8f7f2e3c9ab63231903774a206bf4eb6c134730322977c7}]
xpack.encryptedSavedObjects.encryptionKey: 64619ebf2744058edb0fb150afd78f2a
xpack.reporting.encryptionKey: ee244ef15dbd337fb5b3b8b6ef85ad42
xpack.security.encryptionKey: d79546a5a33e6a1dc514b7b05604ec49