Kibana unavailable

Elastic Stack Kibana
1

Hi,

I am running into an issue: the kibana is not available. When I check the log it gives me the error that kibana version is not compliant with the version of elasticsearch. It looks like I have to downgrade it in order to make it work.
I am wondering if downgrading kibana may delete available dashboards, it is minor release 7.17 to 7.16?

Many thanks

2

Please share your Kibana logs.

3

hi,
Thank you for your reply.
There is no need for logs sharing here. The question is theoretical:

  • if I downgrade Kibana to a minor release, will I lose the dashboards that I have previously saved?

Many thanks

4

Without seeing what is the error in the log you are getting is hard to tell what could happens.

But you need to check if your Kibana is in the same version as Elasticsearch, they need to be in the same minor version, for example, if you are running Elasticsearch 7.17.3, you need Kibana 7.17.3.

5

Hi,
Thank you for your reply.
.
From startup.log today I get this:

FATAL  Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Unexpected Elasticsearch ResponseError: statusCode: 400, method: PUT, url: /.kibana_7.16.3_reindex_temp?wait_for_active_shards=all&timeout=60s error: [illegal_argument_exception]: Validation Failed: 1: this action would add [2] shards, but this cluster currently has [4000]/[4000] maximum normal shards open;,


 FATAL  Error: Unable to complete saved object migrations for the [.kibana_task_manager] index. Please check the health of your Elasticsearch cluster and try again. Unexpected Elasticsearch ResponseError: statusCode: 400, method: PUT, url: /.kibana_task_manager_7.16.3_reindex_temp?wait_for_active_shards=all&timeout=60s error: [illegal_argument_exception]: Validation Failed: 1: this action would add [2] shards, but this cluster currently has [4000]/[4000] maximum normal shards open;,

From kibana.log I get the following:

"type":"log","@timestamp":"2022-10-13T13:56:30+00:00","tags":["warning","plugins","security","config"],"pid":9902,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2022-10-13T13:56:30+00:00","tags":["warning","plugins","reporting","config"],"pid":9902,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2022-10-13T13:56:30+00:00","tags":["warning","plugins","encryptedSavedObjects"],"pid":9902,"message":"Saved objects encryption key is not set.This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2022-10-13T13:56:30+00:00","tags":["warning","plugins","actions"],"pid":9902,"message":"APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2022-10-13T13:56:30+00:00","tags":["warning","plugins","alerting"],"pid":9902,"message":"APIs are disabled because the Encrypted Saved Objectsplugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2022-10-13T13:56:30+00:00","tags":["info","plugins","ruleRegistry"],"pid":9902,"message":"Installing common resources shared between all indices"}
{"type":"log","@timestamp":"2022-10-13T13:56:31+00:00","tags":["info","plugins","reporting","config"],"pid":9902,"message":"Chromium sandbox provides an additional layer of protection, and is supported for Linux Amazon Linux 2018.03 OS. Automatically enabling Chromium sandbox."}
{"type":"log","@timestamp":"2022-10-13T13:56:31+00:00","tags":["error","elasticsearch-service"],"pid":9902,"message":"This version of Kibana (v7.17.6) is incompatible with the following Elasticsearch nodes in your cluster: v7.16.3 @ 127.0.0.1:9200 (10.116.39.168), v7.16.3 @ 10.116.37.201:9200 (10.116.37.201), v7.16.3 @ 10.116.38.68:9200 (10.116.38.68), v7.16.3 @ 10.116.39.197:9200 (10.116.39.197), v7.16.3 @ 127.0.0.1:9200 (10.116.37.151)"}
{"type":"log","@timestamp":"2022-10-13T14:16:30+00:00","tags":["error","plugins","ruleRegistry"],"pid":9902,"message":"Error: Timeout: it took more than 1200000ms\n    atTimeout._onTimeout (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:54:20)\n    at listOnTimeout (node:internal/timers:559:17)\n    at processTimers (node:internal/timers:502:7)"}
{"type":"log","@timestamp":"2022-10-13T14:16:30+00:00","tags":["error","plugins","ruleRegistry"],"pid":9902,"message":"Error: Failure installing common resources shared between all indices. Timeout: it took more than 1200000ms\n    at ResourceInstaller.installWithTimeout (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:63:13)\n    at ResourceInstaller.installCommonResources (/usr/share/kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.js:77:5)"}

Many thanks

6

Your elasticsearch cluster is running 7.16.3, you need to use Kibana 7.16.3.

You will need to downgrade it, maybe you will need to remove it from the system and install again.

You also need to check later why this version mismatch happened.

Your dashboards are stored as saved objects in elasticsearch, so probably there will be no impact on them.

7

Thank you.
I was just wondering if it is not better to upgrade the ES instead?

8

Fixing your Kibana version issue is easier than upgrade 5 elasticsearch nodes.

You should fix the version mismatch and after that you can plan your upgrade.

Also, from the logs you shared, it seems that you have other issues in your cluster as well:

this action would add [2] shards, but this cluster currently has [4000]/[4000] maximum normal shards open

This means that you may have too many shards, you also need to solve this before upgrading elasticsearch and having a working Kibana would help you to manage your cluster an fix issues.

9

Hi,
Thank you for your reply , I really appreciate your help.
Even though the Kibana is installed and running, I cannot find the Kibana index (kibana-int) that should contain some important queries. The cluster was put into production by other person, and unfortunately there is no much documentation avaiable on this topic.

I understood that kibana dashboard stored and elasticsearch cluster kibana-int. Is this setting can be updated in any config file , in such case what is the file name where I can find the corresponding configuration?

The other issue is probably related to memory, since my cluster went down today for no space left on the device. Adding an additional node would solve the issue?

Many thanks!

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.