Snapshot & Restore failing while restoring indices from ES 1.6.1 to ES 2.1.2

Elastic Stack Elasticsearch
#1

Hey all,

I'm upgrading our ES cluster from 1.6.1 to 2.1.2, we have spun up new cluster with ES 2.1.2
I have reindexed documents on ES 1.6.1 so that it passes all the checks on ES migration plugin. Now when I create a snapshot on ES 1.6.1 and try to restore it on 2.1.2 it throws an error

{"error":{"root_cause":[{"type":"snapshot_restore_exception","reason":"[stage-05-2016:05-03_1215] cannot restore index [stage-index-1] because it cannot be upgraded"}],"type":"snapshot_restore_exception","reason":"[stage-05-2016:05-03_1215] cannot restore index [stage-index-1] because it cannot be upgraded","caused_by":{"type":"illegal_state_exception","reason":"unable to upgrade the mappings for the index [stage-index-1], reason: [Mapper for [_id] conflicts with existing mapping in other types:\n[mapper [_id] has different [index] values]]","caused_by":{"type":"illegal_argument_exception","reason":"Mapper for [_id] conflicts with existing mapping in other types:\n[mapper [_id] has different [index] values]"}}},"status":500}

I believe this is because of .percolator type document which indexes _id by default in ES 1.6.1 even though if you don't specify explicitly in the mapping. I don't know if this has anything to do with the .percolator type documents.

Note: Snapshot and Restore works perfectly fine if I remove .percolator type mapping.

Any help would be really appreciated!

Let me know if any other information is required.

#2

Sounds like the migration plugin should have catch that.

Could you open an issue on Github elasticsearch-migration repo with the mappings for your index?

Thanks!

#3

Thanks for the reply @dadoonet
I have filed an issue: https://github.com/elastic/elasticsearch-migration/issues/56
But it still doesn't seem to solve the problem, even though I create a mapping for .percolator type without _id field on ES 1.6.1; elasticsearch creates one.
For example:
My .percolator mapping

{ 
    ".percolator" : {
    "_routing" : {
      "required" : true
    },
    "_timestamp" : {
      "enabled" : true
    },
	"properties" : {
			"percolator_id"       : { "type" : "string", "index" : "not_analyzed", "include_in_all" : false },
			"oid"                 : { "type" : "integer" },
			"user_id"             : { "type" : "integer" },

			"query"               : { "type" : "object", "enabled" : false }
		}
	}
}

ES 1.6.1 creates a mapping with:

{
  "stage-index-1": {
    "mappings": {
      ".percolator": {
        "_id": {
          "index": "not_analyzed"
        },
        "_routing": {
          "required": true
        },
        "_timestamp": {
          "enabled": true
        },
        "properties": {
          "oid": {
            "type": "integer",
            "doc_values": true
          },
          "percolator_id": {
            "type": "string",
            "index": "not_analyzed",
            "include_in_all": false
          },
          "query": {
            "type": "object",
            "enabled": false
          },
          "user_id": {
            "type": "integer"
          }
        }
      }
    }
  }
}

Note the _id being created in the mapping.
So my guess is because _id is created by default for .percolator documents in earlier versions it is preventing me to restore snapshot to ES 2.1.2 version.

Please let me know if there is any workaround.

Elasticsearch dedicated percolator doc values in percolator script
#4

+1
I was able to reproduce the same behavior around _id getting automatically created in .percolator type, which blocks us from migrating up.

1 Like
#5

@dadoonet @warkolm any input on this issue would be really helpful!

#6

Thanks for help.
We have fixed this by doing Snapshot & Restore without .percolator documents and wrote a script to manually migrate those documents so that we can at-least go ahead and upgrade elasticsearch.

1 Like
#7