Cluster wont start after upgrade from 5.6 to 6.3.1/6.3.2

[matejzero]

Hello,

I upgraded our ELK cluster from 5.6.9 to 6.3.1 last week. I check upgrade assistant before I started the upgrade and it gave a GO to upgrade. It was a rolling upgrade and most of the things went fine (and because of the rolling restart, I probably didn't notice todays problem).

Today I had to replace SSL certs and did a full cluster restart and now it's not coming back up because I have different settings for _all fields in different types. This are of course 5.6 indices, which should still work in 6.x.

This is a production cluster and I'm really out of ideas on what to do. I searched this forum and github, but either I'm the only one with the problem or my google fu is failing!

Any ideas?

Error from log:

[2018-08-03T17:26:09,213][ERROR][o.e.g.GatewayMetaState   ] [es4] failed to read local state, exiting...
java.lang.IllegalStateException: unable to upgrade the mappings for the index [[logs-httpd-2018.07-1/dajLHrjeR--7xuOzt0ow0g]]
	at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.checkMappingsCompatibility(MetaDataIndexUpgradeService.java:196) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.upgradeIndexMetaData(MetaDataIndexUpgradeService.java:95) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:245) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:89) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.node.Node.<init>(Node.java:432) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.node.Node.<init>(Node.java:252) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.3.2.jar:6.3.2]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.3.2.jar:6.3.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) [elasticsearch-6.3.2.jar:6.3.2]
Caused by: java.lang.IllegalArgumentException: Mapper for [_all] conflicts with existing mapping in other types:
[mapper [_all] has different [norms] values, cannot change from disable to enabled, mapper [_all] is used by multiple types. Set update_all_types to true to update [omit_norms] across all types.]
	at org.elasticsearch.index.mapper.FieldTypeLookup.checkCompatibility(FieldTypeLookup.java:130) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.index.mapper.FieldTypeLookup.copyAndAddAll(FieldTypeLookup.java:94) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:439) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:356) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:305) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:284) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.checkMappingsCompatibility(MetaDataIndexUpgradeService.java:192) ~[elasticsearch-6.3.2.jar:6.3.2]
	... 15 more

This is only a part of the message, since I'm limited to 7k characters. Full log at https://pastebin.com/7jKkktCz

[matejzero]

With the help of IRC, me and dakrone came to the same idea of manually erasing the indice from hard disk and start the cluster again.

Since this was an ES6 cluster, folders didn't have indice name, but a hash, which was dajLHrjeR--7xuOzt0ow0g (es reported it in log file) in my example.

I erased all folders with this name, started elasticsearch and it came up. I lost the index, but there was only a half-a-day of data in it and I can live with that. Or reindex if there will be a need.