Kibana server is not ready yet with 503 status but the service is running

I've installed the products version 6.8.4 both Kibana and Elasticsearch on Ubuntu 18.04 on two nodes (Kibana only installed on 1 node). However when I start the Kibana server, the errors are produce as log below

{"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:infra@6.8.4","info"],"pid":16330,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:rollup@6.8.4","info"],"pid":16330,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:remote_clusters@6.8.4","info"],"pid":16330,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:cross_cluster_replication@6.8.4","info"],"pid":16330,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:translations@6.8.4","info"],"pid":16330,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:upgrade_assistant@6.8.4","info"],"pid":16330,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:uptime@6.8.4","info"],"pid":16330,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:oss_telemetry@6.8.4","info"],"pid":16330,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:metrics@6.8.4","info"],"pid":16330,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:timelion@6.8.4","info"],"pid":16330,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:elasticsearch@6.8.4","info"],"pid":16330,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["license","warning","xpack"],"pid":16330,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [invalid_index_name_exception] Invalid index name [_xpack], must not start with '_'., with { index_uuid=\"_na_\" & index=\"_xpack\" } :: {\"path\":\"/_xpack\",\"statusCode\":400,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"invalid_index_name_exception\\\",\\\"reason\\\":\\\"Invalid index name [_xpack], must not start with '_'.\\\",\\\"index_uuid\\\":\\\"_na_\\\",\\\"index\\\":\\\"_xpack\\\"}],\\\"type\\\":\\\"invalid_index_name_exception\\\",\\\"reason\\\":\\\"Invalid index name [_xpack], must not start with '_'.\\\",\\\"index_uuid\\\":\\\"_na_\\\",\\\"index\\\":\\\"_xpack\\\"},\\\"status\\\":400}\"}"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:xpack_main@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:graph@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:spaces@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:searchprofiler@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:ml@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:tilemap@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:watcher@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:grokdebugger@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:logstash@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:beats_management@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:maps@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:index_management@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:index_lifecycle_management@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:rollup@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:remote_clusters@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:51Z","tags":["status","plugin:cross_cluster_replication@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from yellow to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2020-08-21T09:30:52Z","tags":["reporting","warning"],"pid":16330,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"} {"type":"log","@timestamp":"2020-08-21T09:30:52Z","tags":["status","plugin:reporting@6.8.4","error"],"pid":16330,"state":"red","message":"Status changed from uninitialized to red - X-Pack plugin is not installed on the [data] Elasticsearch cluster.","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2020-08-21T09:30:52Z","tags":["info","task_manager"],"pid":16330,"message":"Installing .kibana_task_manager index template version: 6080499."} {"type":"log","@timestamp":"2020-08-21T09:30:52Z","tags":["info","task_manager"],"pid":16330,"message":"Installed .kibana_task_manager index template: version 6080499 (API version 1)"} {"type":"log","@timestamp":"2020-08-21T09:31:21Z","tags":["license","warning","xpack"],"pid":16330,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [invalid_index_name_exception] Invalid index name [_xpack], must not start with '_'., with { index_uuid=\"_na_\" & index=\"_xpack\" } :: {\"path\":\"/_xpack\",\"statusCode\":400,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"invalid_index_name_exception\\\",\\\"reason\\\":\\\"Invalid index name [_xpack], must not start with '_'.\\\",\\\"index_uuid\\\":\\\"_na_\\\",\\\"index\\\":\\\"_xpack\\\"}],\\\"type\\\":\\\"invalid_index_name_exception\\\",\\\"reason\\\":\\\"Invalid index name [_xpack], must not start with '_'.\\\",\\\"index_uuid\\\":\\\"_na_\\\",\\\"index\\\":\\\"_xpack\\\"},\\\"status\\\":400}\"}"} {"type":"log","@timestamp":"2020-08-21T09:31:22Z","tags":["warning","maps"],"pid":16330,"message":"Error scheduling telemetry task, received Request Timeout after 30000ms"}

about the X-pack, one of my Elasticsearch maybe no longer has a valid license anymore. So I wanted to uninstall x-pack on the Kibana too. But I'm not sure if this is a root cause of the error.

I've tried to uninstall Kibana x-pack plugin but it said Unable to remove plugin because of error: "You are using the standard distribution of Kibana. Please install the OSS-only distribution to remove X-Pack features."

I don't see any OSS-only distribution here. Where should I find it?

Btw, here're the information from my node

{ "name" : "x.x.x.x", "cluster_name" : "Test Cluster@Elassandra", "cluster_uuid" : "ff68e8ea-4d3a-4403-80d1-55028c85f9cd", "version" : { "number" : "6.8.4", "build_flavor" : "oss", "build_type" : "deb", "build_hash" : "bcb22b9", "build_date" : "2020-07-29T08:30:21.424869Z", "build_snapshot" : false, "lucene_version" : "7.7.2", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" }

It looks like your Elasticsearch nodes are the OSS version, so it makes sense that none of the X-Pack features are present.

You should be able to download the OSS Kibana. I don't see a direct link to them from the download page but if you just grab the link to the version you need and add -oss after kibana in the filename, that artifact should be there. For example https://artifacts.elastic.co/downloads/kibana/kibana-oss-6.8.4-linux-x86_64.tar.gz

1 Like

Add -oss between the link is works! thanks.

And now my server is running Kibana-oss 6.8.4. The warning from X-pack is gone but the error ''Kibana server is not ready yet" still persists.

Kibana log itself doesn't tell much just only this
{"type":"log","@timestamp":"2020-08-24T04:42:12Z","tags":["fatal","root"],"pid":12305,"message":"{ Error: Request Timeout after 30000ms\n at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15\n at Timeout.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:384:7)\n at ontimeout (timers.js:436:11)\n at tryOnTimeout (timers.js:300:5)\n at listOnTimeout (timers.js:263:5)\n at Timer.processTimers (timers.js:223:10)\n status: undefined,\n displayName: 'RequestTimeout',\n message: 'Request Timeout after 30000ms',\n body: undefined }"}

So I've no idea for the next action. please suggest.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.