Kibana 6.7 service trying to create new index after each newstart, but timing out each time

#1

I'm on 6.7 and this issue seems to follow me since I made the update.

After each newstart of the kibana service, it tries to migrate the .kibana index from .index_ to .index_<n+1>. This is odd, and it would have not been noticed if I hadn't made so much maintenance on the machine those past few weeks. But this wouldn't be as annoying if it didn't timed-out and provoking a restart of the service due to fatal error, each time, as I had right after the upgrade. Because it times-out and the service becomes a new PID, I presume the migration is locked under the old PID and it can't go on, throwing me an exception like the following:

Apr 18 15:37:08 es1 kibana[18180]: {"type":"log","@timestamp":"2019-04-18T13:37:08Z","tags":["warning","migrations"],"pid":18180,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_7 and restarting Kibana."}

The worse is, even if I delete the old index and makes an alias to the new index created by the migration process, I get now:

Index .kibana_10 belongs to a version of Kibana that cannot be automatically migrated. Reset it or use the X-Pack upgrade assistant

I tried to delete all .kibana index, so the index would be generated from scratch but I comes back to the original problem:

May 17 14:19:14 es1 kibana[9612]: {"type":"log","@timestamp":"2019-05-17T12:19:14Z","tags":["info","migrations"],"pid":9612,"message":"Creating index .kibana_1."}
May 17 14:19:44 es1 kibana[9612]: {"type":"log","@timestamp":"2019-05-17T12:19:44Z","tags":["status","plugin:spaces@6.7.1","error"],"pid":9612,"state":"red","message":"Status changed from yellow to red - Request Timeout after 30000ms","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
May 17 14:19:44 es1 kibana[9612]: {"type":"log","@timestamp":"2019-05-17T12:19:44Z","tags":["warning"],"pid":9612,"message":"Error loading maps telemetry: Error: Request Timeout after 30000ms"}
May 17 14:19:44 es1 kibana[9612]: {"type":"log","@timestamp":"2019-05-17T12:19:44Z","tags":["fatal","root"],"pid":9612,"message":"{ Error: Request Timeout after 30000ms\n    at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:355:15\n    at Timeout.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:384:7)\n    at ontimeout (timers.js:436:11)\n    at tryOnTimeout (timers.js:300:5)\n    at listOnTimeout (timers.js:263:5)\n    at Timer.processTimers (timers.js:223:10)\n  status: undefined,\n  displayName: 'RequestTimeout',\n  message: 'Request Timeout after 30000ms',\n  body: undefined,\n  isBoom: true,\n  isServer: true,\n  data: null,\n  output:\n   { statusCode: 503,\n     payload:\n      { statusCode: 503,\n        error: 'Service Unavailable',\n        message: 'Request Timeout after 30000ms' },\n     headers: {} },\n  reformat: [Function],\n  [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/esUnavailable' }"}
May 17 14:19:44 es1 kibana[9612]:  FATAL  Error: Request Timeout after 30000ms
May 17 14:19:44 es1 systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
May 17 14:19:44 es1 systemd[1]: kibana.service: Unit entered failed state.
May 17 14:19:44 es1 systemd[1]: kibana.service: Failed with result 'exit-code'.
May 17 14:19:45 es1 systemd[1]: kibana.service: Service hold-off time over, scheduling restart.
May 17 14:19:45 es1 systemd[1]: Stopped Kibana.
May 17 14:19:45 es1 systemd[1]: Started Kibana.

And then when it tries to create the index again, it tells me anothe Kibana migrate the index already. Again.

Isn't there a way to see where the index creation/migration is stuck? It can't be performance anymore, as I made some changes so elasticsearch could have access to more RAM and I already see the effect.

Maybe there is a way to change the timeout, maybe give it 1 or 2 minutes?

Is it a normal behaviour anyway, that it tries to migrate the existing index at each service restart?

(Tyler Smalley) #2

Hey, let my first try to clarify some things.

In Kibana 6.5 we added index migrations to Kibana. This allows us to make changes to the underlying mappings of the Kibana index without requiring users to do so. With the upgrade to 6.x, we had to remove mapping types from the index, and this had to be done by the user in the Upgrade Assistant or manually. There is more information about troubleshooting migrations here and technical details here.

For the migrations, we use an alias which allows us to swap out the old index for the new of the names you pointed out (.kibana_X).

What version are you coming from? The error you mentioned belongs to a version of Kibana that cannot be automatically migrated. Reset it or use the X-Pack upgrade assistant leads me to believe you're coming from a pre-6.0 index, which has multiple types and needs to be migrated prior to upgrading.

The logs you posted after deleting all the Kibana data were 30 second timeouts to Elasticsearch. Is it possible that ES was not up or accessible during that time?

#3

I originally installed a 5.x which was updated to 6.2 then 6.3 and finally 6.7

No according to _cat/health, status was green, and even looking at the elasticsearch task manager I could see that it was actively and timely processing bulk operation. Also I had recently added some resources to the server hosting the Elastic Stack, as I had some issue with performances, but I still got the timeout in this specific case.

Now, as I will soon be on holiday leave, I wanted to have the system up in running before I went, so I reinstalled the kibana (which is now 6.7.2 instead of 6.7.1) and it seems to work now.

BUT as an update to 7 is planned to be done in the next few months, I would like to be sure that something like doesn't happen again. Do I still have to worry? Is there any test I can do to be sure?

(Tyler Smalley) #4

Sorry we couldn't get to the bottom of the issue. Going forward you really shouldn't have issues since Kibana manages the migrations itself prior to startup. One thing you could do it snapshot/restore the index, or re-index into another name and attempt the upgrade prior.