Hello,
I upgraded our ELK cluster from 5.6.9 to 6.3.1 last week. I check upgrade assistant before I started the upgrade and it gave a GO to upgrade. It was a rolling upgrade and most of the things went fine (and because of the rolling restart, I probably didn't notice todays problem).
Today I had to replace SSL certs and did a full cluster restart and now it's not coming back up because I have different settings for _all
fields in different types. This are of course 5.6 indices, which should still work in 6.x.
This is a production cluster and I'm really out of ideas on what to do. I searched this forum and github, but either I'm the only one with the problem or my google fu is failing!
Any ideas?
Error from log:
[2018-08-03T17:26:09,213][ERROR][o.e.g.GatewayMetaState ] [es4] failed to read local state, exiting...
java.lang.IllegalStateException: unable to upgrade the mappings for the index [[logs-httpd-2018.07-1/dajLHrjeR--7xuOzt0ow0g]]
at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.checkMappingsCompatibility(MetaDataIndexUpgradeService.java:196) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.upgradeIndexMetaData(MetaDataIndexUpgradeService.java:95) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:245) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:89) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.node.Node.<init>(Node.java:432) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.node.Node.<init>(Node.java:252) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.3.2.jar:6.3.2]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.3.2.jar:6.3.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) [elasticsearch-6.3.2.jar:6.3.2]
Caused by: java.lang.IllegalArgumentException: Mapper for [_all] conflicts with existing mapping in other types:
[mapper [_all] has different [norms] values, cannot change from disable to enabled, mapper [_all] is used by multiple types. Set update_all_types to true to update [omit_norms] across all types.]
at org.elasticsearch.index.mapper.FieldTypeLookup.checkCompatibility(FieldTypeLookup.java:130) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.index.mapper.FieldTypeLookup.copyAndAddAll(FieldTypeLookup.java:94) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:439) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:356) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:305) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:284) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.checkMappingsCompatibility(MetaDataIndexUpgradeService.java:192) ~[elasticsearch-6.3.2.jar:6.3.2]
... 15 more
This is only a part of the message, since I'm limited to 7k characters. Full log at https://pastebin.com/7jKkktCz