7.0 Upgrade Assistant

Elastic Stack Elasticsearch
1

Hello World!

I'm running Elastic Stack 6.6.1 and per Kibana's 7.0 Upgrade Assistant (Management->7.0 Upgrade Assistant), I'm seeing following:

  • Cluster
    • GCS Repository settings changed (Documentation) - warning
      • nodes with repository-gcs installed: [XXXX]
  • Indicies
    • Index created before 6.0 (Documentation) - critical
      • Index: .security-6
      • Details: this index was created using version: 5.6.4

Unfortunately both Documentation links are broken (probably will be up later).

Did I miss Upgrading internal indices for 6.x step or ... ?

Please advise.

2

Hi @alexus,

From the error message looks like index created before 6.0 have not been upgraded.

I am assuming your current cluster (on version 6.6.1) was upgraded from 5.6.4.
If you have followed the step then you can check user authentication works with native realm user.
If login succeeds then the step of an upgrade from 5.6.4 was done, else you will have to
Upgrade internal indices from 6.x step before proceeding with 7.0.
Hope this helps.

Regards,
Yogesh Gaikwad

3

I thought bellow part was performed:

POST /_xpack/migration/upgrade/.security

yet, Upgrade Assistant thinks otherwise...

output from above POST:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index",
        "index_uuid" : "_na_",
        "index" : ".security"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index",
    "index_uuid" : "_na_",
    "index" : ".security"
  },
  "status" : 404
}

I believe .security is alias, yet actual index is .security-6...

4

Hi @alexus,

Not sure why it says no security index found,
could you please check response for GET /_xpack/migration/assistance ?
This should list all the internal indices like .security, .watcher?

Could you also please attach the logs that you are getting when you do the upgrade?
This would help us debug further.

Regards,
Yogesh Gaikwad

5

Hi @alexus,

I think we have uncovered a bug wherein if the shard allocation is not enabled and you try to do the upgrade internal indices step, you get into a situation where the .security index gets deleted.
The bulk response was not checked and even when there were failures, we continued with the deletion of old index assuming reindex was successful. This is a bug and I can reproduce this.

On your setup, while following the upgrade guide this is what happened:-

  1. Disable shard allocation before the upgrade
  2. Once you upgrade all the nodes, plugins etc. following the document you need to enable shard allocation (Step no. 8)
    I think this is what was missed, to enable shard allocation after restart
  3. Then do the upgrade of internal indices

If you did not do the step of enabling shard allocation, the upgrade fails due to the bug that we uncovered and the .security was deleted.

Thank you for your query and hope this helps. For you to continue on your quest, please enable the cluster shard allocation and then try upgrading the internal indices.

Thanks and Regards,
Yogesh Gaikwad

6

GET /_xpack/migration/assistance

results:

{
  "indices" : { }
}

and

POST /_xpack/migration/upgrade/.security

{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index",
        "index_uuid" : "_na_",
        "index" : ".security"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index",
    "index_uuid" : "_na_",
    "index" : ".security"
  },
  "status" : 404
}

other then above output, elasticsearch master node does not produce any errors...

Reenable shard allocation. is enabled, yet upgrade fails(

7

Hi,

I think there may be multiple issues at play here, but one thing to note is that the _xpack/migration APIs will not be used to prepare for an upgrade to 7.0 - we're still in the process of updating all of our documentation here, but going forward, this will primarily be done through Kibana's Upgrade Assistant rather than the Migration APIs. However, the Kibana functionality to automate reindexing and upgrades of internal indices will not be available until version 6.7 of the Elastic stack.

The broken documentation links are a known issue - the location of that documentation has changed since the release of 6.6, and will be corrected in the next release. In the meantime, you can find the correct documentation here.

8

Hi @alexus,

I think once the .security gets deleted I do not think reenabling shard allocation on the upgraded setup would work, here I am assuming that you tried that on your upgraded setup. (As the .security index has been deleted there is nothing upgrade)
You could try from the 5.6.4 backup and retry the upgrade process to follow the steps.

I tried this on my setup while upgrading from 5.6.14 version 6.6.0 version where, before I did the upgrade of .security I reenabled shard allocation. If I did this I could see migration is successful but if I do not enable it then the .security index is deleted thereby failing any future upgrade tries.

Hope this helps.

Thanks and Regards,
Yogesh Gaikwad

9

@Yogesh_Gaikwad Even though exception from previous comment states otherwise, .security does exists (it is an alias for .security-6 index), so there is something to upgrade after all ...

10

Hi @alexus,

I think I missed your comment.
When you say the .security-6 index exists, are you able to search on it?
Could you do a simple search on .security index and see if there is anything in the index?
GET /.security/_search

As I mentioned earlier if the upgrade failed (when the shards were not enabled) you may not be able to recover from the current state. I would suggest starting again with from old cluster from backup and to see if following the steps helps you.

Regards,
Yogesh Gaikwad

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.