Another Kibana instance appears to be migrating the index

Jambo80 · January 14, 2021, 3:41pm

Hi

I am relatively new to ELK and recently upgraded from 6.X to 6.8.13. When I start Kibana I now see the following error:

"tags":["warning","migrations"],"pid":15615,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_2 and restarting Kibana."}`

If I look at the .kibana indices:

curl -XGET http://myserver.com:9200/_cat/indices | grep kibana

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 10201 100 10201 0 0 33260 0 --:--:-- --:--:-
green open .kibana_1 vwMLshKwSMuPQYI20kiB_g 1 1 101 0 445.6kb 222.8kb
green open .kibana_2 M9ngJUYbS3ua1yD4Rkrg9A 1 1 0 0 522b 261b
green open .kibana e0dyMNN7Qb6JQ1aIFfnuPQ 1 1 101 1 540.6kb 270.3kb

I see no aliases:

curl -XGET http://myserver.com:9200/_cat/aliases

Nothing Returned

I have stopped kibana, deleted .kibana_2 and restarted kibana serveral times and the same problem persists. However I can see no other instance of kibana running.

I do not have a snapshot and I am trying to get kibana back whilst retaining the many dashboards and visualisations within.

Can anyone advise from looking at the sizes of the .kibana and .kibana_1 indices if I am right that it appears to have copied the majority of information over?

Is it possible that I can manually create the .kibana alias for .kibana_1 without deleting the original .kibana index? such as

curl -X POST "myserver.com:9200/_aliases" -H 'Content-Type: application/json' -d' { "actions" : [ { "add" : { "index" : ".kibana_1", "alias" : ".kibana" } } ] }'

Any other suggestions of how to solve this problem?

Extremely grateful for any advice

flash1293 · January 15, 2021, 8:12am

Hey, it's possible something is going wrong during the migration if there are corrupted saved objects (this can happen for a bunch of reasons). Please stop Kibana, delete the .kibana_2 index again, restart Kibana and check the logs for error messages. It's likely there are some related to "migration" - please copy/paste them here.

Jambo80 · January 15, 2021, 10:06am

Hi @flash1293 sorry just noticed that Kibana log location is not configured. I will switch them on, repeat the process and post again

Jambo80 · January 15, 2021, 2:28pm

Hi @flash1293,

I stopped kibana deleted both kibana_1 and kibana_2 and restarted kibana. All the logs related to "migrations" are below

[root@myserver.com data]# cat /var/log/kibana/kibana.stdout | grep migrations

{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["info","migrations"],"pid":76501,"message":"Creating index .kibana_2."}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["info","migrations"],"pid":76501,"message":"Reindexing .kibana to .kibana_1"}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["info","migrations"],"pid":76557,"message":"Creating index .kibana_2."}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["warning","migrations"],"pid":76557,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_2 and restarting Kibana."}

Please let me know if these are helpful in diagnosing the problem or if you need to see more complete log output.

Much appreciated

flash1293 · January 15, 2021, 3:09pm

This is not including the part I'm after, could you paste the full log after the last restart?

Jambo80 · January 15, 2021, 4:04pm

Hi @flash1293

The full log is too big to post here unless there is a method to attach a log file?

Are these any good or is there a way I can filter the log so that i reduce the noise and don't miss any information you require?

>SNIP
{"type":"log","@timestamp":"2021-01-15T14:19:10Z","tags":["reporting","debug","exportTypes"],"pid":76501,"message":"Found exportType at /usr/share/kibana/node_modules/x-pack/plugins/reporting/export_types/csv/server/index.js"}
{"type":"log","@timestamp":"2021-01-15T14:19:10Z","tags":["reporting","debug","exportTypes"],"pid":76501,"message":"Found exportType at /usr/share/kibana/node_modules/x-pack/plugins/reporting/export_types/png/server/index.js"}
{"type":"log","@timestamp":"2021-01-15T14:19:10Z","tags":["reporting","debug","exportTypes"],"pid":76501,"message":"Found exportType at /usr/share/kibana/node_modules/x-pack/plugins/reporting/export_types/printable_pdf/server/index.js"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["server","uuid","uuid"],"pid":76501,"message":"Resuming persistent Kibana instance UUID: a2ad2177-7495-4106-89a6-e306f4854118"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["plugin","debug"],"pid":76501,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["status","plugin:elasticsearch@6.8.13","info"],"pid":76501,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["license","debug","xpack"],"pid":76501,"message":"Calling [data] Elasticsearch _xpack API. Polling frequency: 30001"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["debug","upgrade_assistant","reindex_worker"],"pid":76501,"message":"Starting worker..."}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["debug","upgrade_assistant","reindex_worker"],"pid":76501,"message":"Polling for reindex operations"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["license","info","xpack"],"pid":76501,"message":"Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["monitoring-ui","es-client"],"pid":76501,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["status","plugin:xpack_main@6.8.13","info"],"pid":76501,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["status","plugin:graph@6.8.13","info"],"pid":76501,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["status","plugin:searchprofiler@6.8.13","info"],"pid":76501,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["status","plugin:ml@6.8.13","info"],"pid":76501,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["status","plugin:tilemap@6.8.13","info"],"pid":76501,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["status","plugin:watcher@6.8.13","info"],"pid":76501,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["status","plugin:grokdebugger@6.8.13","info"],"pid":76501,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["status","plugin:logstash@6.8.13","info"],"pid":76501,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["status","plugin:beats_management@6.8.13","info"],"pid":76501,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["status","plugin:index_management@6.8.13","info"],"pid":76501,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["status","plugin:index_lifecycle_management@6.8.13","info"],"pid":76501,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["status","plugin:rollup@6.8.13","info"],"pid":76501,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["status","plugin:remote_clusters@6.8.13","info"],"pid":76501,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["status","plugin:cross_cluster_replication@6.8.13","info"],"pid":76501,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["info","monitoring-ui","kibana-monitoring"],"pid":76501,"message":"Starting monitoring stats collection"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["license","debug","xpack"],"pid":76501,"message":"Calling [monitoring] Elasticsearch _xpack API. Polling frequency: 30001"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["status","plugin:security@6.8.13","info"],"pid":76501,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["status","plugin:maps@6.8.13","info"],"pid":76501,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["debug","stats-collection"],"pid":76501,"message":"All collectors are not ready. Waiting for 60000ms longer."}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":76501,"message":"Skipping bulk uploading because not all collectors are ready"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":76501,"message":"Resetting lastFetchWithUsage because not all collectors are ready"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["license","info","xpack"],"pid":76501,"message":"Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["reporting","browser-driver","warning"],"pid":76501,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["reporting","browser-driver","debug"],"pid":76501,"message":"Browser installed at /var/lib/kibana/headless_shell-linux/headless_shell"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["reporting","warning"],"pid":76501,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["reporting","debug"],"pid":76501,"message":"Browser type: chromium"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["reporting","debug"],"pid":76501,"message":"Chromium sandbox disabled: true"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["status","plugin:reporting@6.8.13","info"],"pid":76501,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["reporting","worker","debug"],"pid":76501,"message":"CSV: Registering CSV worker"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["reporting","esqueue","worker","debug"],"pid":76501,"message":"kjydb5jk1n11d9f95dfme87t - Created worker for job type csv"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["reporting","worker","debug"],"pid":76501,"message":"PNG: Registering PNG worker"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["reporting","esqueue","worker","debug"],"pid":76501,"message":"kjydb5jl1n11d9f95dfn131b - Created worker for job type PNG"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["reporting","worker","debug"],"pid":76501,"message":"PDF: Registering PDF worker"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["reporting","esqueue","worker","debug"],"pid":76501,"message":"kjydb5jm1n11d9f95dfngwh8 - Created worker for job type printable_pdf"}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["reporting","debug"],"pid":76501,"message":"Running on os \"linux\", distribution \"Centos\", release \"7.9.2009\""}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["debug","task_manager"],"pid":76501,"message":"Not installing .kibana_task_manager index template: version 6081399 already exists."}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["debug","task_manager"],"pid":76501,"message":"Not installing .kibana_task_manager index template: version 6081399 already exists."}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["debug","task_manager"],"pid":76501,"message":"Not installing .kibana_task_manager index template: version 6081399 already exists."}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["info","migrations"],"pid":76501,"message":"Creating index .kibana_2."}
{"type":"log","@timestamp":"2021-01-15T14:19:11Z","tags":["info","migrations"],"pid":76501,"message":"Reindexing .kibana to .kibana_1"}
{"type":"log","@timestamp":"2021-01-15T14:19:12Z","tags":["debug","legacy-proxy"],"pid":76501,"message":"\"getConnections\" has been called."}
{"type":"log","@timestamp":"2021-01-15T14:19:12Z","tags":["debug","upgrade_assistant","reindex_worker"],"pid":76501,"message":"Could not fetch riendex operations from Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:12Z","tags":["debug","root"],"pid":76501,"message":"shutting root down"}
{"type":"log","@timestamp":"2021-01-15T14:19:13Z","tags":["debug","config"],"pid":76557,"message":"Marking config path as handled: server"}
{"type":"log","@timestamp":"2021-01-15T14:19:13Z","tags":["debug","http"],"pid":76557,"message":"registering route handler for [/core]"}
{"type":"log","@timestamp":"2021-01-15T14:19:13Z","tags":["debug","root"],"pid":76557,"message":"starting root"}
{"type":"log","@timestamp":"2021-01-15T14:19:13Z","tags":["debug","config"],"pid":76557,"message":"Marking config path as handled: logging"}
{"type":"log","@timestamp":"2021-01-15T14:19:14Z","tags":["debug","server"],"pid":76557,"message":"starting server"}
<SNIP

Jambo80 · January 15, 2021, 4:08pm

Further...

>SNIP
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["reporting","debug","exportTypes"],"pid":76557,"message":"Found exportType at /usr/share/kibana/node_modules/x-pack/plugins/reporting/export_types/csv/server/index.js"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["reporting","debug","exportTypes"],"pid":76557,"message":"Found exportType at /usr/share/kibana/node_modules/x-pack/plugins/reporting/export_types/png/server/index.js"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["reporting","debug","exportTypes"],"pid":76557,"message":"Found exportType at /usr/share/kibana/node_modules/x-pack/plugins/reporting/export_types/printable_pdf/server/index.js"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["server","uuid","uuid"],"pid":76557,"message":"Resuming persistent Kibana instance UUID: a2ad2177-7495-4106-89a6-e306f4854118"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["plugin","debug"],"pid":76557,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["status","plugin:elasticsearch@6.8.13","info"],"pid":76557,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["license","debug","xpack"],"pid":76557,"message":"Calling [data] Elasticsearch _xpack API. Polling frequency: 30001"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["debug","upgrade_assistant","reindex_worker"],"pid":76557,"message":"Starting worker..."}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["debug","upgrade_assistant","reindex_worker"],"pid":76557,"message":"Polling for reindex operations"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["license","info","xpack"],"pid":76557,"message":"Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["monitoring-ui","es-client"],"pid":76557,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["status","plugin:xpack_main@6.8.13","info"],"pid":76557,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["status","plugin:graph@6.8.13","info"],"pid":76557,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["status","plugin:searchprofiler@6.8.13","info"],"pid":76557,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["status","plugin:ml@6.8.13","info"],"pid":76557,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["status","plugin:tilemap@6.8.13","info"],"pid":76557,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["status","plugin:watcher@6.8.13","info"],"pid":76557,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["status","plugin:grokdebugger@6.8.13","info"],"pid":76557,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["status","plugin:logstash@6.8.13","info"],"pid":76557,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["status","plugin:beats_management@6.8.13","info"],"pid":76557,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["status","plugin:index_management@6.8.13","info"],"pid":76557,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["status","plugin:index_lifecycle_management@6.8.13","info"],"pid":76557,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["status","plugin:rollup@6.8.13","info"],"pid":76557,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["status","plugin:remote_clusters@6.8.13","info"],"pid":76557,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["status","plugin:cross_cluster_replication@6.8.13","info"],"pid":76557,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["info","monitoring-ui","kibana-monitoring"],"pid":76557,"message":"Starting monitoring stats collection"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["license","debug","xpack"],"pid":76557,"message":"Calling [monitoring] Elasticsearch _xpack API. Polling frequency: 30001"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["status","plugin:security@6.8.13","info"],"pid":76557,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["status","plugin:maps@6.8.13","info"],"pid":76557,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["debug","stats-collection"],"pid":76557,"message":"All collectors are not ready. Waiting for 60000ms longer."}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":76557,"message":"Skipping bulk uploading because not all collectors are ready"}
{"type":"log","@timestamp":"2021-01-15T14:19:18Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":76557,"message":"Resetting lastFetchWithUsage because not all collectors are ready"}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["license","info","xpack"],"pid":76557,"message":"Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active"}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["reporting","browser-driver","warning"],"pid":76557,"message":"Enabling the Chromium sandbox provides an additional layer of protection."}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["reporting","browser-driver","debug"],"pid":76557,"message":"Browser installed at /var/lib/kibana/headless_shell-linux/headless_shell"}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["reporting","warning"],"pid":76557,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["reporting","debug"],"pid":76557,"message":"Browser type: chromium"}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["reporting","debug"],"pid":76557,"message":"Chromium sandbox disabled: true"}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["status","plugin:reporting@6.8.13","info"],"pid":76557,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["reporting","worker","debug"],"pid":76557,"message":"CSV: Registering CSV worker"}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["reporting","esqueue","worker","debug"],"pid":76557,"message":"kjydbbju1n2ld9f95dc2tl8v - Created worker for job type csv"}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["reporting","worker","debug"],"pid":76557,"message":"PNG: Registering PNG worker"}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["reporting","esqueue","worker","debug"],"pid":76557,"message":"kjydbbjv1n2ld9f95dc3hfqx - Created worker for job type PNG"}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["reporting","worker","debug"],"pid":76557,"message":"PDF: Registering PDF worker"}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["reporting","esqueue","worker","debug"],"pid":76557,"message":"kjydbbjw1n2ld9f95dc3ygo0 - Created worker for job type printable_pdf"}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["reporting","debug"],"pid":76557,"message":"Running on os \"linux\", distribution \"Centos\", release \"7.9.2009\""}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["debug","task_manager"],"pid":76557,"message":"Not installing .kibana_task_manager index template: version 6081399 already exists."}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["debug","task_manager"],"pid":76557,"message":"Not installing .kibana_task_manager index template: version 6081399 already exists."}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["debug","task_manager"],"pid":76557,"message":"Not installing .kibana_task_manager index template: version 6081399 already exists."}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["info","migrations"],"pid":76557,"message":"Creating index .kibana_2."}
{"type":"log","@timestamp":"2021-01-15T14:19:19Z","tags":["warning","migrations"],"pid":76557,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_2 and restarting Kibana."}
<SNIP

rudolf · January 21, 2021, 9:12am

6.8 migrations are almost identical to the way they work in 7.2 so this documentation would be good background to read through https://www.elastic.co/guide/en/kibana/7.2/upgrade-migrations.html#upgrade-migrations-process To prevent migrations from failing again make sure you read the section on "preventing migration failures" before you retry the migration.

It looks like you upgraded from <= 6.5 when Kibana still used a .kibana index, this will prevent you from creating any aliases. If this is the case, .kibana will still have all your data and I would recommend the following:

Shutdown all Kibana instances
Make a backup snapshot of all your data
Delete .kibana_1 and .kibana_2
Start a single Kibana v6.8 instance and make sure you capture all the logs from the first time Kibana starts up. (starting multiple nodes in parallel is supported, it just makes debugging harder)

Jambo80 · January 26, 2021, 3:27pm

Hi @rudolf and @flash1293

I was <6.5 before upgrade as suggested.

I confirmed I had only one instance of Kibana running "ps guax | grep -i kibana" and followed your suggested steps above. The logs showed nothing more than I had already posted and the same situation continued to arise .

In case its useful to anybody else here is what I did to fix it:

reindex .kibana and .kibana_1 so I had a backup (thanks to @Kjetil_Larsen)
compared the json for .kibana and .kibana_1 (they were identical)
stop Kibana.service
delete .kibana and .kibana_2

curl -XDELETE http://myserver.com:9200/.kibana
curl -XDELETE http://myserver.com:9200/.kibana_2

create the alias for .kibana_1

curl -XPOST "myserver.com:9200/_aliases" -H 'Content-Type: application/json' -d' { "actions" : [ { "add" : { "index" : ".kibana_1", "alias" : ".kibana" } } ] }'

start Kibana.service

Kibana is operational again

Appreciate your assistance.

system · February 23, 2021, 3:27pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.