"Format version is not supported"

Elastic Stack Elasticsearch
1

Hi,

I upgraded from Elasticsearch 2.1.1 to 5.1.1, then by mistake down to 2.1.1 again, and back up to 5.1.1.

Now, two of my indices are out of services, with reason below. What can I do to recover?

$ curl -XGET 'http://localhost:9200/_cluster/allocation/explain?pretty' -d'{  "index": "logstash-2016.12.14",  "shard": 1,  "primary": true}'
{
  "shard" : {
    "index" : "logstash-2016.12.14",
    "index_uuid" : "78iFiCffRAKSBu7uf4Ekuw",
    "id" : 1,
    "primary" : true
  },
  "assigned" : false,
  "shard_state_fetch_pending" : false,
  "unassigned_info" : {
    "reason" : "CLUSTER_RECOVERED",
    "at" : "2016-12-14T13:37:20.867Z",
    "delayed" : false,
    "allocation_status" : "no_valid_shard_copy"
  },
  "allocation_delay_in_millis" : 60000,
  "remaining_delay_in_millis" : 0,
  "nodes" : {
    "H4UxrgYbSzmcT2UE3WCJnw" : {
      "node_name" : "H4UxrgY",
      "node_attributes" : { },
      "store" : {
        "shard_copy" : "CORRUPT",
        "store_exception" : "CorruptIndexException[failed engine (reason: [corrupt file (source: [start])]) (resource=preexisting_corruption)]; nested: IOException[failed engine (reason: [corrupt file (source: [start])])]; nested: IndexFormatTooNewException[Format version is not supported (resource MMapIndexInput(path=\"/usr/share/elasticsearch/data/elasticsearch/nodes/0/indices/78iFiCffRAKSBu7uf4Ekuw/1/index/_5xh.cfs\") [slice=_5xh.dim]): 3 (needs to be between 0 and 2)]; "
      },
      "final_decision" : "NO",
      "final_explanation" : "the copy of the shard is corrupt",
      "weight" : 36.25,
      "decisions" : [ ]
    }
  }
}
2

Bad luck here. :frowning:

Apart trying again to have all your cluster running 5.0.1 I'm not sure how else you can recover from this situation.
I let other people commenting but worse case I'd delete this index.
Apparently it's about logs logstash-2016.12.14 so may be you can afford it without reindexing?

BTW how many nodes do you have? Is node H4UxrgY a 5.0 one? I guess so judging by its name.

3

Hi, Thanks for your help. Yes, there is only the one node in the cluster. I'm not sure if the index is really corrupted, it just says it has the wrong version (too high). Is there no way to "force" it to be allocated with the newest version?

4

Sorry. I don't know.

May @jpountz has an idea?

5

This version check is used by Lucene to detect when users try to read an index with an older version than the one that was used to create it.

Version 3 of the BKD tree maps to https://issues.apache.org/jira/browse/LUCENE-7501 which was added to Lucene 6.3, and Elasticsearch 5.1.1 is the first version to use this version of Lucene.

Are you sure you are running 5.1.1 and not an older 5.0 release? The reason I am asking is that you would get this error if you were on 5.0, but not on 5.1.

What do GET /_cat/nodes?v&h=id,name,v and GET logstash-2016.12.14/_settings return?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.