Kibana upgrade from 7.12.0 to 7.13.1 is failing

Elastic Stack Kibana
1

Hi,

I have been trying to upgrade kibana for a few days ow and it does not work. I have the following error :

{"type":"log","@timestamp":"2021-06-11T10:10:39+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-06-11T10:10:39+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2021-06-11T10:10:39+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 19ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:39+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana] WAIT_FOR_YELLOW_SOURCE -> SET_SOURCE_WRITE_BLOCK. took: 17ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:39+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana_task_manager] INIT -> WAIT_FOR_YELLOW_SOURCE. took: 38ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:39+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana_task_manager] WAIT_FOR_YELLOW_SOURCE -> SET_SOURCE_WRITE_BLOCK. took: 4ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:40+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana] SET_SOURCE_WRITE_BLOCK -> CREATE_REINDEX_TEMP. took: 371ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:40+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana_task_manager] SET_SOURCE_WRITE_BLOCK -> CREATE_REINDEX_TEMP. took: 421ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:41+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana_task_manager] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP. took: 1628ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:41+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP -> REINDEX_SOURCE_TO_TEMP_WAIT_FOR_TASK. took: 7ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:41+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana] CREATE_REINDEX_TEMP -> REINDEX_SOURCE_TO_TEMP. took: 1801ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:41+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP -> REINDEX_SOURCE_TO_TEMP_WAIT_FOR_TASK. took: 11ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:42+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana_task_manager] REINDEX_SOURCE_TO_TEMP_WAIT_FOR_TASK -> SET_TEMP_WRITE_BLOCK. took: 733ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:42+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana_task_manager] SET_TEMP_WRITE_BLOCK -> CLONE_TEMP_TO_TARGET. took: 289ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:44+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana_task_manager] CLONE_TEMP_TO_TARGET -> OUTDATED_DOCUMENTS_SEARCH. took: 1803ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:44+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH -> UPDATE_TARGET_MAPPINGS. took: 20ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:44+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 324ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:45+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> MARK_VERSION_INDEX_READY. took: 223ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:45+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana_task_manager] MARK_VERSION_INDEX_READY -> DONE. took: 421ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:45+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana_task_manager] Migration completed after 5912ms"}
{"type":"log","@timestamp":"2021-06-11T10:10:48+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana] REINDEX_SOURCE_TO_TEMP_WAIT_FOR_TASK -> SET_TEMP_WRITE_BLOCK. took: 6954ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:49+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana] SET_TEMP_WRITE_BLOCK -> CLONE_TEMP_TO_TARGET. took: 201ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:49+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana] CLONE_TEMP_TO_TARGET -> OUTDATED_DOCUMENTS_SEARCH. took: 21ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:49+00:00","tags":["info","savedobjects-service"],"pid":27709,"message":"[.kibana] OUTDATED_DOCUMENTS_SEARCH -> OUTDATED_DOCUMENTS_TRANSFORM. took: 230ms."}
{"type":"log","@timestamp":"2021-06-11T10:10:49+00:00","tags":["error","savedobjects-service"],"pid":27709,"message":"TypeError: embeddables.getEnhancement(...).migrations[version] is not a function\n    at /usr/share/kibana/src/plugins/embeddable/common/lib/migrate.js:30:91\n    at Array.forEach (<anonymous>)\n    at Object.migrate (/usr/share/kibana/src/plugins/embeddable/common/lib/migrate.js:28:31)\n    at /usr/share/kibana/src/plugins/dashboard/server/saved_objects/dashboard_migrations.js:187:45\n    at Array.forEach (<anonymous>)\n    at /usr/share/kibana/src/plugins/dashboard/server/saved_objects/dashboard_migrations.js:181:10\n    at tryTransformDoc

On the ElastisSearch side I get this error :

[2021-06-11T10:41:44,399][INFO ][o.e.t.LoggingTaskListener] [node-1] 1235458005 finished with response BulkByScrollResponse[took=180.9ms,timed_out=false,sliceId=null,updated=0,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[{"index":".kibana_7.13.1_reindex_temp","type":"_doc","id":"monitoring:lens-ui-telemetry:777b7820-825a-11eb-bd40-f3b1b94afb2f","cause":{"type":"cluster_block_exception","reason":"index [.kibana_7.13.1_reindex_temp] blocked by: [FORBIDDEN/8/index write (api)];"},"status":403}, {"index":".kibana_7.13.1_reindex_temp","type":"_doc","id":"monitoring:lens-ui-telemetry:7d718030-825a-11eb-bd40-f3b1b94afb2f","cause":{"type":"cluster_block_exception","reason":"index [.kibana_7.13.1_reindex_temp] blocked by: [FORBIDDEN/8/index write (api)];"},"status":403}, {"index":".kibana_7.13.1_reindex_temp","type":"_doc","id":"monitoring:lens-ui-telemetry:8381c700-825a-11eb-bd40-f3b1b94afb2f","cause":{"type":"cluster_block_exception","reason":"index [.kibana_7.13.1_reindex_temp] blocked by: [FORBIDDEN/8/index write (api)];"},"status":403}],search_failures=[]]
[2021-06-11T10:41:44,431][INFO ][o.e.t.LoggingTaskListener] [node-1] 1235458008 finished with response BulkByScrollResponse[took=219.4ms,timed_out=false,sliceId=null,updated=25,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]

Do you have any idea of what is going on?

Thank you in advance

EDIT : ElasticSearch is already 7.13.1

2

The matter is not time sensitive as I was able to rollback to 7.12.0 but I would really like to be able to upgrade kibana.

3

From your second log message from ELK it tells me that .kibana_7.13.1_reindex_temp has a problem

now that you have kibana running at 7.12.

go and delete that .kibana_7.13.1_reindex_temp index and try out.

4

I did that but it goes back to the same error. Could it be something wrong with the 7.12 index?

5

it is definatly something wrong with index. I had same exact error on one of the index.
Backup that 7.12 index and delete it. if it does something weired you can restore it. make sure you have proper restore procedure down from command line. as it kibana stops working you have to do it from command line.

6

@Vincent_Uribe - Welcome to our community forum!

This is a known issue with Kibana version 7.13.1 where the migration of saved objects fails with dashboard panels that are by value and have drilldowns (c.f Known issue).

For now, it is best to wait for Kibana 7.13.2 to be released.

Let us know if you need further clarifications. Thank you! :slight_smile:

1 Like
7

Oh at least I know I’m not crazy :sweat_smile:…
Thank you for your quick response…

1 Like
8

@Vincent_Uribe - FYI - Version 7.13.2 is available (c.f Elastic Stack 7.13.2 released) :slight_smile:

9

Awesome! Thank you, we will give it a spin :grin:

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.