Kibana server is not ready yet with 503 status but the service is running

Elastic Stack Kibana
1

I've installed the products version 6.8.4 both Kibana and Elasticsearch on Ubuntu 18.04 on two nodes (Kibana only installed on 1 node). However when I start the Kibana server, the errors are produce as log below

{"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:infra@6.8.4","info"],"pid":16330,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:rollup@6.8.4","info"],"pid":16330,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:remote_clusters@6.8.4","info"],"pid":16330,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:cross_cluster_replication@6.8.4","info"],"pid":16330,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:translations@6.8.4","info"],"pid":16330,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:upgrade_assistant@6.8.4","info"],"pid":16330,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:uptime@6.8.4","info"],"pid":16330,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:oss_telemetry@6.8.4","info"],"pid":16330,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:metrics@6.8.4","info"],"pid":16330,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:timelion@6.8.4","info"],"pid":16330,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:elasticsearch@6.8.4","info"],"pid":16330,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["license","warning","xpack"],"pid":16330,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [invalid_index_name_exception] Invalid index name [_xpack], must not start with '_'., with { index_uuid=\"_na_\" & index=\"_xpack\" } :: {\"path\":\"/_xpack\",\"statusCode\":400,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"invalid_index_name_exception\\\",\\\"reason\\\":\\\"Invalid index name [_xpack], must not start with '_'.\\\",\\\"index_uuid\\\":\\\"_na_\\\",\\\"index\\\":\\\"_xpack\\\"}],\\\"type\\\":\\\"invalid_index_name_exception\\\",\\\"reason\\\":\\\"Invalid index name [_xpack], must not start with '_'.\\\",\\\"index_uuid\\\":\\\"_na_\\\",\\\"index\\\":\\\"_xpack\\\"},\\\"status\\\":400}\"}"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:xpack_main@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:graph@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:spaces@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:searchprofiler@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:ml@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:tilemap@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:watcher@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:grokdebugger@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:logstash@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:beats_management@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:maps@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:index_management@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:index_lifecycle_management@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:rollup@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:remote_clusters@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:cross_cluster_replication@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:52Z","tags":["reporting","warning"],"pid":16330,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"} {"type":"log","@timestamp":"2020-08-21T09:30:52Z","tags":["status","plugin:reporting@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from uninitialized to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:52Z","tags":["info","task_manager"],"pid":16330,"message":"Installing .kibana_task_manager index template version: 6080499."} {"type":"log","@timestamp":"2020-08-21T09:30:52Z","tags":["info","task_manager"],"pid":16330,"message":"Installed .kibana_task_manager index template: version 6080499 (API version 1)"} {"type":"log","@timestamp":"2020-08-21T09:31:21Z","tags":["license","warning","xpack"],"pid":16330,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [invalid_index_name_exception] Invalid index name [_xpack], must not start with '_'., with { index_uuid=\"_na_\" & index=\"_xpack\" } :: {\"path\":\"/_xpack\",\"statusCode\":400,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"invalid_index_name_exception\\\",\\\"reason\\\":\\\"Invalid index name [_xpack], must not start with '_'.\\\",\\\"index_uuid\\\":\\\"_na_\\\",\\\"index\\\":\\\"_xpack\\\"}],\\\"type\\\":\\\"invalid_index_name_exception\\\",\\\"reason\\\":\\\"Invalid index name [_xpack], must not start with '_'.\\\",\\\"index_uuid\\\":\\\"_na_\\\",\\\"index\\\":\\\"_xpack\\\"},\\\"status\\\":400}\"}"} {"type":"log","@timestamp":"2020-08-21T09:31:22Z","tags":["warning","maps"],"pid":16330,"message":"Error scheduling telemetry task, received Request Timeout after 30000ms"}

about the X-pack, one of my Elasticsearch maybe no longer has a valid license anymore. So I wanted to uninstall x-pack on the Kibana too. But I'm not sure if this is a root cause of the error.

I've tried to uninstall Kibana x-pack plugin but it said Unable to remove plugin because of error: "You are using the standard distribution of Kibana. Please install the OSS-only distribution to remove X-Pack features."

I don't see any OSS-only distribution here. Where should I find it?

Btw, here're the information from my node

{ "name" : "x.x.x.x", "cluster_name" : "Test Cluster@Elassandra", "cluster_uuid" : "ff68e8ea-4d3a-4403-80d1-55028c85f9cd", "version" : { "number" : "6.8.4", "build_flavor" : "oss", "build_type" : "deb", "build_hash" : "bcb22b9", "build_date" : "2020-07-29T08:30:21.424869Z", "build_snapshot" : false, "lucene_version" : "7.7.2", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" }

2

It looks like your Elasticsearch nodes are the OSS version, so it makes sense that none of the X-Pack features are present.

You should be able to download the OSS Kibana. I don't see a direct link to them from the download page but if you just grab the link to the version you need and add -oss after kibana in the filename, that artifact should be there. For example https://artifacts.elastic.co/downloads/kibana/kibana-oss-6.8.4-linux-x86_64.tar.gz

1 Like
3

Add -oss between the link is works! thanks.

And now my server is running Kibana-oss 6.8.4. The warning from X-pack is gone but the error ''Kibana server is not ready yet" still persists.

Kibana log itself doesn't tell much just only this
{"type":"log","@timestamp":"2020-08-24T04:42:12Z","tags":["fatal","root"],"pid":12305,"message":"{ Error: Request Timeout after 30000ms\n at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15\n at Timeout.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:384:7)\n at ontimeout (timers.js:436:11)\n at tryOnTimeout (timers.js:300:5)\n at listOnTimeout (timers.js:263:5)\n at Timer.processTimers (timers.js:223:10)\n status: undefined,\n displayName: 'RequestTimeout',\n message: 'Request Timeout after 30000ms',\n body: undefined }"}

So I've no idea for the next action. please suggest.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.