Timeout while migrating .kibana-6 index to .kibana_7

Elastic Stack Kibana
1

I'm currently updating my Elastic Stack from 6.3.2 to 6.7. For some reason, each time I start the kibana service after update, there is a timeout during the original migration.

Apr 18 15:36:36 es1 kibana[18082]: {"type":"log","@timestamp":"2019-04-18T13:36:36Z","tags":["warning"],"pid":18082,"message":"Error loading maps telemetry: Error: Request Timeout after 30000ms"}
Apr 18 15:36:36 es1 kibana[18082]: {"type":"log","@timestamp":"2019-04-18T13:36:36Z","tags":["fatal","root"],"pid":18082,"message":"{ Error: Request Timeout after 30000ms\n    at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:384:7)\n    at ontimeout (timers.js:436:11)\n    at tryOnTimeout (timers.js:300:5)\n    at listOnTimeout (timers.js:263:5)\n    at Timer.processTimers (timers.js:223:10)\n  status: undefined,\n  displayName: 'RequestTimeout',\n  message: 'Request Timeout after 30000ms',\n  body: undefined,\n  isBoom: true,\n  isServer: true,\n  data: null,\n  output:\n   { statusCode: 503,\n     payload:\n      { statusCode: 503,\n        error:  'Service Unavailable',\n        message: 'Request Timeout after 30000ms' },\n     headers: {} },\n  reformat: [Function],\n  [Symbol(SavedObjectsClientError)]: 'SavedObjectsClient/esUnavailable' }"}
Apr 18 15:36:36 es1 kibana[18082]:  FATAL  Error: Request Timeout after 30000ms
Apr 18 15:36:37 es1 systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Apr 18 15:36:37 es1 systemd[1]: kibana.service: Unit entered failed state.
Apr 18 15:36:37 es1 systemd[1]: kibana.service: Failed with result 'exit-code'.
Apr 18 15:36:37 es1 systemd[1]: kibana.service: Service hold-off time over, scheduling restart.
Apr 18 15:36:37 es1 systemd[1]: Stopped Kibana.
Apr 18 15:36:37 es1 systemd[1]: Started Kibana.

Following that when it tries to migrate the kibana index again, it says another Kibana service uses the .kibana_7 index.

Apr 18 15:36:47 es1 kibana[18180]: {"type":"log","@timestamp":"2019-04-18T13:36:47Z","tags":["info","migrations"],"pid":18180,"message":"Creating index .kibana_7."}
Apr 18 15:37:08 es1 kibana[18180]: {"type":"log","@timestamp":"2019-04-18T13:37:08Z","tags":["warning","migrations"],"pid":18180,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_7 and restarting Kibana."}

Is there a way to solve this way, or at least a way for me to do that manually while the kibana service is stoped?

2

I ran into same issue, I solved it by:

  1. Ensuring Kibana was not running
  2. Deleting the new index (as indicated in your log message ".kibana_7")
  3. Removing the read-only flag from old index
  4. Restarting new kibana

Curl commands I ran (ensure to update URL and index names as needed!:

# This first one will show you all .kibana indexes if you want to check them
curl -XGET -H 'Content-Type: application/json' http://elasticsearch-client:9200/.kibana*/
curl -XDELETE -H 'Content-Type: application/json' http://elasticsearch-client:9200/.kibana_7/
curl -XPUT -H 'Content-Type: application/json' http://elasticsearch-client:9200/.kibana_6/_settings -d '{ "index.blocks.read_only_allow_delete": null }'

Good luck!

3

Unfortunately it stills went to timeout.

For reference here is my .kibana-6 index settings:

"settings": {
  "index": {
    "refresh_interval": "60s",
    "number_of_shards": "1",
    "provided_name": ".kibana-6",
    "mapper": {
      "dynamic": "false"
    },
    "creation_date": "1529418443216",
    "number_of_replicas": "0",
    "uuid": "hwazyWGVQ0uw_1gXd87wLQ",
    "version": {
      "created": "5061099",
      "upgraded": "6070199"
    }
  }
}
4

Hmmm... Read your kibana logs closely, especially first few dozen lines after starting; it gave me good hints. I'm no expert, so not sure I can offer much more advice (this was my first Upgrade).

5

What I ended up doing is following:

I remove the .kibana-6 index

curl -XDELETE -H 'Content-Type: application/json' http://localhost:9200/.kibana-6/

I then created an alias to .kibana to the .kibana_7 index

curl -X POST "localhost:9200/_aliases" -H 'Content-Type: application/json' -d' { "actions" : [ { "add" : { "index" : ".kibana_7", "alias" : ".kibana" } } ] }'

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.