Unassigned shards after upgrade due to different _all values

[matejzero]

I upgraded elasticsearch from 5.6 to 6.3 today and 4 shards stayed in unassigned state (it's replicas that is).

Looking at the reasons, it looks like it bugs him that I have different settings for _all fields in different document types (this index was created with ES 5.6).

I tried setting number_of_replicas to 0 and changing mapping and set _all fields to disable, but I keep getting the same error:

{
  "error": {
    "root_cause": [
      {
        "type": "remote_transport_exception",
        "reason": "[es4][x.x.x.x:9300][indices:admin/mapping/put]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Mapper for [_all] conflicts with existing mapping in other types:\n[mapper [_all] has different [norms] values, cannot change from disable to enabled]"
  },
  "status": 400
}

I also closed one of the indexes and when I try to open it, I get this:

{
  "error": {
    "root_cause": [
      {
        "type": "exception",
        "reason": "Failed to verify index [logs-httpd-2018.07-1/dajLHrjeR--7xuOzt0ow0g]"
      }
    ],
    "type": "exception",
    "reason": "Failed to verify index [logs-httpd-2018.07-1/dajLHrjeR--7xuOzt0ow0g]",
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "Mapper for [_all] conflicts with existing mapping in other types:\n[mapper [_all] has different [norms] values, cannot change from disable to enabled]"
    }
  },
  "status": 500
}

Indice mapping: https://pastebin.com/td7JBGy2

I googled around and didn't find a solution for my problem.

What are my options here?
Is it possible to save the index?
Should I reindex?

[matejzero]

Due to some other problems, I had to rebuild one of the nodes in the cluster. After the node was up, I was still seeing the same problems.

I checked the state with _cluster/allocation/explainendpoint, which told me that reroute failed due to too many failed attempts. It said in the response that I could retry the reroute with _cluster/reroute?retry_failed=true. That did the trick for all online indices.

I also have a closed indice, which was previously giving me the same problem, which I can't open.

When calling logs-httpd-2018.07-1/_open api, I get the response:

{
  "error": {
    "root_cause": [
      {
        "type": "remote_transport_exception",
        "reason": "[es2][x.x.x.x:9300][indices:admin/open]"
      }
    ],
    "type": "illegal_state_exception",
    "reason": "unable to upgrade the mappings for the index [[logs-httpd-2018.07-1/dajLHrjeR--7xuOzt0ow0g]]",
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "Mapper for [_all] conflicts with existing mapping in other types:\n[mapper [_all] is used by multiple types. Set update_all_types to true to update [omit_norms] across all types.]"
    }
  },
  "status": 500
}

I also tried changing mappings, but cant.