[SOLVED] Kibana crashes on reindex after 6.5.0 upgrade

Elastic Stack Kibana
1

Was on 6.4, upgraded elastic 6.5 & restarted nodes - > upgrade complete.
Upgraded kibana 6.4 -> 6.5.0 and restarted-> crash an burn

I retried several times after deleting .kibana_1 and .kibana_2

  log   [12:06:46.987] [info][status][plugin:beats_management@6.5.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [12:06:47.024] [info][status][plugin:apm@6.5.0] Status changed from uninitialized to green - Ready
  log   [12:06:47.200] [info][status][plugin:canvas@6.5.0] Status changed from uninitialized to green - Ready
  log   [12:06:47.207] [info][status][plugin:console@6.5.0] Status changed from uninitialized to green - Ready
  log   [12:06:47.209] [info][status][plugin:console_extensions@6.5.0] Status changed from uninitialized to green - Ready
  log   [12:06:47.213] [info][status][plugin:notifications@6.5.0] Status changed from uninitialized to green - Ready
  log   [12:06:47.253] [info][status][plugin:infra@6.5.0] Status changed from uninitialized to green - Ready
  log   [12:06:47.262] [info][status][plugin:metrics@6.5.0] Status changed from uninitialized to green - Ready
  log   [12:06:47.270] [info][status][plugin:elasticsearch@6.5.0] Status changed from yellow to green - Ready
  log   [12:06:47.619] [warning][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml
  log   [12:06:47.621] [info][status][plugin:reporting@6.5.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [12:06:47.669] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active
  log   [12:06:47.674] [info][status][plugin:xpack_main@6.5.0] Status changed from yellow to green - Ready
  log   [12:06:47.675] [info][status][plugin:searchprofiler@6.5.0] Status changed from yellow to green - Ready
  log   [12:06:47.676] [info][status][plugin:ml@6.5.0] Status changed from yellow to green - Ready
  log   [12:06:47.677] [info][status][plugin:tilemap@6.5.0] Status changed from yellow to green - Ready
  log   [12:06:47.677] [info][status][plugin:watcher@6.5.0] Status changed from yellow to green - Ready
  log   [12:06:47.678] [info][status][plugin:index_management@6.5.0] Status changed from yellow to green - Ready
  log   [12:06:47.678] [info][status][plugin:rollup@6.5.0] Status changed from yellow to green - Ready
  log   [12:06:47.679] [info][status][plugin:graph@6.5.0] Status changed from yellow to green - Ready
  log   [12:06:47.681] [info][status][plugin:grokdebugger@6.5.0] Status changed from yellow to green - Ready
  log   [12:06:47.682] [info][status][plugin:logstash@6.5.0] Status changed from yellow to green - Ready
  log   [12:06:47.682] [info][status][plugin:beats_management@6.5.0] Status changed from yellow to green - Ready
  log   [12:06:47.683] [info][status][plugin:reporting@6.5.0] Status changed from yellow to green - Ready
  log   [12:06:47.683] [info][kibana-monitoring][monitoring-ui] Starting monitoring stats collection
  log   [12:06:47.697] [info][status][plugin:security@6.5.0] Status changed from yellow to green - Ready
  log   [12:06:47.761] [info][license][xpack] Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active
  log   [12:06:48.374] [info][migrations] Creating index .kibana_2.
  log   [12:06:48.809] [info][migrations] Reindexing .kibana to .kibana_1
 error  [12:06:49.523] [fatal][root] [resource_not_found_exception] task [k3s35GjtR7ySZkTMIiumgQ:776425] isn't running and hasn't stored its results :: {"path":"/_tasks/k3s35GjtR7ySZkTMIiumgQ%3A776425","query":{},"statusCode":404,"response":"{\"error\":{\"root_cause\":[{\"type\":\"resource_not_found_exception\",\"reason\":\"task [k3s35GjtR7ySZkTMIiumgQ:776425] isn't running and hasn't stored its results\"}],\"type\":\"resource_not_found_exception\",\"reason\":\"task [k3s35GjtR7ySZkTMIiumgQ:776425] isn't running and hasn't stored its results\"},\"status\":404}"}
    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)
    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)
    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)
    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4949:19)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)

 FATAL  [resource_not_found_exception] task [k3s35GjtR7ySZkTMIiumgQ:776425] isn't running and hasn't stored its results :: {"path":"/_tasks/k3s35GjtR7ySZkTMIiumgQ%3A776425","query":{},"statusCode":404,"response":"{\"error\":{\"root_cause\":[{\"type\":\"resource_not_found_exception\",\"reason\":\"task [k3s35GjtR7ySZkTMIiumgQ:776425] isn't running and hasn't stored its results\"}],\"type\":\"resource_not_found_exception\",\"reason\":\"task [k3s35GjtR7ySZkTMIiumgQ:776425] isn't running and hasn't stored its results\"},\"status\":404}"}

Ideas?
I tried manually reindexing the .kibana to a custom name which works fine:

curl -X POST http://10.10.20.201:9200/_reindex -H 'Content-Type: application/json' -d @- << EOF
{
"source": {
"index": ".kibana"
},
"dest": {
"index": ".kibana_kristofer"
}
}
EOF

2

There is a breaking change after an upgrade to 6.5, have you checked the release notes? (https://www.elastic.co/guide/en/kibana/current/release-notes-6.5.0.html#known-issues-6.5.0).

Are you facing the same issues?

Thanks
Rashmi

1 Like
3

Ah, I did give that a quick scan but it didn't seem applicable since security is a paid and I'm not paying.

It does seem related but since I don't have security enabled there are plenty of steps I can't perform... :open_mouth:

4

Are the ES logs available during this time?

5

How many ES nodes in the cluster do you have?

6

I have 2 nodes.

Aaahhh, checked the elasticsearch logs and found:
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

I've had similar issues before so I removed read-only from all indices:

curl -X PUT http://localhost:9200/_all/_settings -H 'Content-Type: application/json' -d'{ "index.blocks.read_only_allow_delete" : false } }'

And now kibana finishes migration!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.