Upgrade es 6.2 to 6.3 and kibana doesn't work

Bowen_Liu · September 16, 2019, 9:58am

Hi, all. I managed to upgrade es from es 6.2.4 to 6.3.0 two days ago and I'm now starting to upgrade kibana too.But when I download kibana 6.3.0 and copy old config of kibana 6.2.4 to 6.3.0 and start kibana, it just can't work. I disable the x-pack security of elasticsearch. Here is the kibana config file:

server.port: 5601
server.host: "XX.XXX.XXX.XX"
elasticsearch.url: "http://XX.XXX.XXX.XX:9200"
elasticsearch.requestTimeout: 300000
elasticsearch.startupTimeout: 5000
xpack.monitoring.elasticsearch.url: "http://XX.XXX.XXX.XX:9200"
xpack.security.enabled : false
xpack.monitoring.enabled : true
xpack.graph.enabled : false
xpack.watcher.enabled : false
xpack.ml.enabled : false

and following is the log when I started kibana:

  log   [09:51:56.826] [info][status][plugin:kibana@6.3.0] Status changed from uninitialized to green - Ready
  log   [09:51:56.863] [info][status][plugin:elasticsearch@6.3.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [09:51:56.866] [info][status][plugin:xpack_main@6.3.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [09:51:56.871] [info][status][plugin:searchprofiler@6.3.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [09:51:56.875] [info][status][plugin:tilemap@6.3.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [09:51:56.880] [info][status][plugin:license_management@6.3.0] Status changed from uninitialized to green - Ready
  log   [09:51:56.884] [info][status][plugin:index_management@6.3.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [09:51:57.026] [info][status][plugin:timelion@6.3.0] Status changed from uninitialized to green - Ready
  log   [09:51:57.029] [info][status][plugin:monitoring@6.3.0] Status changed from uninitialized to green - Ready
  log   [09:51:57.031] [info][status][plugin:grokdebugger@6.3.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [09:51:57.033] [info][status][plugin:dashboard_mode@6.3.0] Status changed from uninitialized to green - Ready
  log   [09:51:57.035] [info][status][plugin:logstash@6.3.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [09:51:57.052] [info][status][plugin:apm@6.3.0] Status changed from uninitialized to green - Ready
  log   [09:51:57.058] [info][status][plugin:console@6.3.0] Status changed from uninitialized to green - Ready
  log   [09:51:57.060] [info][status][plugin:console_extensions@6.3.0] Status changed from uninitialized to green - Ready
  log   [09:51:57.063] [info][status][plugin:metrics@6.3.0] Status changed from uninitialized to green - Ready
  log   [09:51:57.809] [warning][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml
  log   [09:51:57.812] [info][status][plugin:reporting@6.3.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [09:51:57.955] [info][listening] Server running at http://10.129.140.70:5601
  log   [09:51:57.969] [info][status][plugin:elasticsearch@6.3.0] Status changed from yellow to green - Ready
  log   [09:51:57.973] [error][status][plugin:xpack_main@6.3.0] Status changed from yellow to red - Red
  log   [09:51:57.973] [error][status][plugin:searchprofiler@6.3.0] Status changed from yellow to red - Red
  log   [09:51:57.974] [error][status][plugin:tilemap@6.3.0] Status changed from yellow to red - Red
  log   [09:51:57.974] [error][status][plugin:index_management@6.3.0] Status changed from yellow to red - Red
  log   [09:51:57.975] [error][status][plugin:grokdebugger@6.3.0] Status changed from yellow to red - Red
  log   [09:51:57.975] [error][status][plugin:logstash@6.3.0] Status changed from yellow to red - Red
  log   [09:51:57.976] [error][status][plugin:reporting@6.3.0] Status changed from yellow to red - Red

The elasticsearch plugin is ok and other plugins aren't, so it doesn't seem like the es problem, and the es is truly ok for I can connect it with cerebro. There is no further log and I don't know how to deal with it. Can someone help please? Thank you so much!!

Brandon_Kobel · September 16, 2019, 5:04pm

Hey @Bowen_Liu, Kibana 6.3.1 includes a bug-fix which potentially solves the issue that you're seeing, but I'd recommend upgrading to 6.3.2 while you're doing so.

When using X-Pack, it was previously possible for Kibana to get stuck in a "red" state, preventing access to the Kibana applications, if Kibana and Elasticsearch were started at the same time. We’ve fixed this issue to ensure that Kibana is always usable when it’s able to talk to Elasticsearch.

From Kibana 6.3.1 | Kibana User Guide [6.3] | Elastic

Bowen_Liu · September 17, 2019, 12:22am

OK，I see. I think that's why the default recommended version is 6.3.2 instead of 6.3.0
I'll try 6.3.2. Thank you very much for your kindly reply.

Bowen_Liu · September 17, 2019, 1:37am

I tried kibana 6.3.2 but still get the wrong status:

log   [01:28:06.460] [info][status][plugin:kibana@6.3.2] Status changed from uninitialized to green - Ready
  log   [01:28:06.497] [info][status][plugin:elasticsearch@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [01:28:06.500] [info][status][plugin:xpack_main@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [01:28:06.505] [info][status][plugin:searchprofiler@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [01:28:06.514] [info][status][plugin:ml@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [01:28:06.553] [info][status][plugin:tilemap@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [01:28:06.555] [info][status][plugin:watcher@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [01:28:06.568] [info][status][plugin:license_management@6.3.2] Status changed from uninitialized to green - Ready
  log   [01:28:06.570] [info][status][plugin:index_management@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [01:28:06.702] [info][status][plugin:timelion@6.3.2] Status changed from uninitialized to green - Ready
  log   [01:28:06.704] [info][status][plugin:graph@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [01:28:06.708] [info][status][plugin:monitoring@6.3.2] Status changed from uninitialized to green - Ready
  log   [01:28:06.710] [info][status][plugin:grokdebugger@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [01:28:06.713] [info][status][plugin:dashboard_mode@6.3.2] Status changed from uninitialized to green - Ready
  log   [01:28:06.715] [info][status][plugin:logstash@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [01:28:06.729] [info][status][plugin:apm@6.3.2] Status changed from uninitialized to green - Ready
  log   [01:28:06.735] [info][status][plugin:console@6.3.2] Status changed from uninitialized to green - Ready
  log   [01:28:06.737] [info][status][plugin:console_extensions@6.3.2] Status changed from uninitialized to green - Ready
  log   [01:28:06.740] [info][status][plugin:metrics@6.3.2] Status changed from uninitialized to green - Ready
  log   [01:28:07.417] [warning][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml
  log   [01:28:07.420] [info][status][plugin:reporting@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [01:28:07.563] [info][listening] Server running at http://10.129.140.70:5601
  log   [01:28:07.629] [info][status][plugin:elasticsearch@6.3.2] Status changed from yellow to green - Ready
  log   [01:28:07.653] [error][status][plugin:xpack_main@6.3.2] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.
  log   [01:28:07.654] [error][status][plugin:searchprofiler@6.3.2] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.
  log   [01:28:07.654] [error][status][plugin:ml@6.3.2] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.
  log   [01:28:07.655] [error][status][plugin:tilemap@6.3.2] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.
  log   [01:28:07.656] [error][status][plugin:watcher@6.3.2] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.
  log   [01:28:07.656] [error][status][plugin:index_management@6.3.2] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.
  log   [01:28:07.657] [error][status][plugin:graph@6.3.2] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.
  log   [01:28:07.657] [error][status][plugin:grokdebugger@6.3.2] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.
  log   [01:28:07.658] [error][status][plugin:logstash@6.3.2] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.
  log   [01:28:07.658] [error][status][plugin:reporting@6.3.2] Status changed from yellow to red - [data] Elasticsearch cluster did not respond with license information.

seems like kibana didn't get the license, when I execute:

curl -XGET "http://10.129.1xx.xx:9200/_xpack?pretty"

I get following info:

{
  "build" : {
    "hash" : "053779d",
    "date" : "2018-07-20T05:25:16.206115Z"
  },
  "license" : null,
  "features" : {
    "graph" : {
      "description" : "Graph Data Exploration for the Elastic Stack",
      "available" : true,
      "enabled" : true
    },
    "logstash" : {
      "description" : "Logstash management component for X-Pack",
      "available" : true,
      "enabled" : true
    },
    "ml" : {
      "description" : "Machine Learning for the Elastic Stack",
      "available" : true,
      "enabled" : true,
      "native_code_info" : {
        "version" : "6.3.2",
        "build_hash" : "903094f295d249"
      }
    },
    "monitoring" : {
      "description" : "Monitoring for the Elastic Stack",
      "available" : true,
      "enabled" : true
    },
    "rollup" : {
      "description" : "Time series pre-aggregation and rollup",
      "available" : true,
      "enabled" : true
    },
    "security" : {
      "description" : "Security for the Elastic Stack",
      "available" : true,
      "enabled" : false
    },
    "watcher" : {
      "description" : "Alerting, Notification and Automation for the Elastic Stack",
      "available" : true,
      "enabled" : true
    }
  },
  "tagline" : "You know, for X"
}

seems that the cluster didn't generate basic license. I execute:

curl -XPOST 'http://10.129.1xx.xx:9200/_xpack/license/start_basic'

and it just blocked. I confirm that the es status is normal and I can execute query from curl. Am I do something wrong? How can I get kibana work up? Thank you!

Bowen_Liu · September 17, 2019, 1:56am

Problem solved.
The new es 6.3.2 cluster truly didn't generate a license which I don't know why. The block reason when I execute:

curl -XPOST 'http://10.129.1xx.xx:9200/_xpack/license/start_basic'

is because the status of the cluster is yellow, seems it will not response for this request when cluster status is not green. I executed this command when the cluster status turned green and I got response. It looks like following:

{"acknowledged":false,"basic_was_started":false,"error_message":"Operation failed: Needs acknowledgement.","acknowledge":{"message":"This license update requires acknowledgement. To acknowledge the license, please read the following messages and call /start_basic again, this time with the \"acknowledge=true\" parameter:","watcher":["Watcher will be disabled"],"logstash":["Logstash will no longer poll for centrally-managed pipelines"],"security":["The following X-Pack security functionality will be disabled: authentication, authorization, ip filtering, and auditing. Please restart your node after applying the license.","Field and document level access control will be disabled.","Custom realms will be ignored."],"monitoring":["Multi-cluster support is disabled for clusters with [BASIC] license. If you are\nrunning multiple clusters, users won't be able to access the clusters with\n[BASIC] licenses from within a single X-Pack Kibana instance. You will have to deploy a\nseparate and dedicated X-pack Kibana instance for each [BASIC] cluster you wish to monitor.","Automatic index cleanup is locked to 7 days for clusters with [BASIC] license."],"graph":["Graph will be disabled"],"ml":["Machine learning will be disabled"],"sql":["JDBC support will be disabled, but you can continue to use SQL CLI and REST endpoint"]}}%

According to the output, I add the param acknowledge=true to the end of the request:

curl -XPOST 'http://10.129.140.72:9200/_xpack/license/start_basic?acknowledge=true'

and finally the license was generated:

{"acknowledged":true,"basic_was_started":true}%

Hope this can help the guys encounter the same problem. Thank you.

system · October 15, 2019, 1:56am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.