Hello,
I have issue with migrate from version 8.19.2 to 9.0.6
On version 8.19.2 from Stack Management/Upgrade assistant I migrated system indices
Next I founded some critical old indexes - deleted them,
And during upgrade I’m receiving below error on stack:
{"@timestamp":"2025-09-13T23:26:01.973Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"elastic-tst-0","elasticsearch.cluster.name":"tst-cluster","error.type":"org.elasticsearch.ElasticsearchException","error.message":"failed to load metadata","error.stack_trace":"org.elasticsearch.ElasticsearchException: failed to load metadata\n\tat org.elasticsearch.server@9.0.6/org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:118)\n\tat org.elasticsearch.server@9.0.6/org.elasticsearch.node.Node.start(Node.java:315)\n\tat org.elasticsearch.server@9.0.6/org.elasticsearch.bootstrap.Elasticsearch.start(Elasticsearch.java:649)\n\tat org.elasticsearch.server@9.0.6/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:446)\n\tat org.elasticsearch.server@9.0.6/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:102)\nCaused by: org.elasticsearch.gateway.CorruptStateException: org.elasticsearch.xcontent.XContentParseException: [-1:22119] [data_stream] failed to parse field [data_stream]\n\tat org.elasticsearch.server@9.0.6/org.elasticsearch.gateway.PersistedClusterStateService.readXContent(PersistedClusterStateService.java:679)\n\tat org.elasticsearch.server@9.0.6/org.elasticsearch.gateway.PersistedClusterStateService.lambda$loadOnDiskState$6(PersistedClusterStateService.java:580)\n\tat org.elasticsearch.server@9.0.6/org.elasticsearch.gateway.PersistedClusterStateService.consumeFromType(PersistedClusterStateService.java:722)\n\tat org.elasticsearch.server@9.0.6/org.elasticsearch.gateway.PersistedClusterStateService.loadOnDiskState(PersistedClusterStateService.java:579)\n\tat org.elasticsearch.server@9.0.6/org.elasticsearch.gateway.PersistedClusterStateService.loadBestOnDiskState(PersistedClusterStateService.java:503)\n\tat org.elasticsearch.server@9.0.6/org.elasticsearch.gateway.PersistedClusterStateService.loadBestOnDiskState(PersistedClusterStateService.java:414)\n\tat org.elasticsearch.server@9.0.6/org.elasticsearch.gateway.GatewayMetaState.createOnDiskPersistedState(GatewayMetaState.java:179)\n\tat org.elasticsearch.server@9.0.6/org.elasticsearch.gateway.GatewayMetaState.createPersistedState(GatewayMetaState.java:147)\n\tat org.elasticsearch.server@9.0.6/org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:105)\n\t... 4 more\nCaused by: org.elasticsearch.xcontent.XContentParseException: [-1:22119] [data_stream] failed to parse field [data_stream]\n\tat org.elasticsearch.xcontent@9.0.6/org.elasticsearch.xcontent.ObjectParser.throwFailedToParse(ObjectParser.java:620)\n\tat org.elasticsearch.xcontent@9.0.6/org.elasticsearch.xcontent.ObjectParser.parseValue(ObjectParser.java:615)\n\tat org.elasticsearch.xcontent@9.0.6/org.elasticsearch.xcontent.ObjectParser.parseSub(ObjectParser.java:633)\n\tat org.elasticsearch.xcontent@9.0.6/org.elasticsearch.xcontent.ObjectParser.parse(ObjectParser.java:316)\n\tat org.elasticsearch.xcontent@9.0.6/org.elasticsearch.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:167)\n\tat org.elasticsearch.server@9.0.6/org.elasticsearch.cluster.metadata.DataStreamMetadata.fromXContent(DataStreamMetadata.java:231)\n\tat org.elasticsearch.xcontent@9.0.6/org.elasticsearch.xcontent.NamedXContentRegistry$Entry.lambda$new$0(NamedXContentRegistry.java:56)\n\tat org.elasticsearch.xcontent@9.0.6/org.elasticsearch.xcontent.NamedXContentRegistry.parseNamedObject(NamedXContentRegistry.java:149)\n\tat org.elasticsearch.xcontent@9.0.6/org.elasticsearch.xcontent.support.AbstractXContentParser.namedObject(AbstractXContentParser.java:432)\n\tat org.elasticsearch.server@9.0.6/org.elasticsearch.cluster.metadata.Metadata$Builder.fromXContent(Metadata.java:2682)\n\tat org.elasticsearch.server@9.0.6/org.elasticsearch.gateway.PersistedClusterStateService.readXContent(PersistedClusterStateService.java:677)\n\t... 12 more\nCaused by: org.elasticsearch.xcontent.XContentParseException: [-1:22093] [data_stream] unknown field [failure_rollover_on_write] did you mean [rollover_on_write]?\n\tat org.elasticsearch.xcontent@9.0.6/org.elasticsearch.xcontent.ObjectParser.lambda$errorOnUnknown$2(ObjectParser.java:106)\n\tat org.elasticsearch.xcontent@9.0.6/org.elasticsearch.xcontent.ObjectParser.parse(ObjectParser.java:300)\n\tat org.elasticsearch.xcontent@9.0.6/org.elasticsearch.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:167)\n\tat org.elasticsearch.server@9.0.6/org.elasticsearch.cluster.metadata.DataStream.fromXContent(DataStream.java:1331)\n\tat org.elasticsearch.server@9.0.6/org.elasticsearch.cluster.metadata.DataStreamMetadata.lambda$static$1(DataStreamMetadata.java:66)\n\tat org.elasticsearch.xcontent@9.0.6/org.elasticsearch.xcontent.ObjectParser.lambda$declareField$10(ObjectParser.java:435)\n\tat org.elasticsearch.xcontent@9.0.6/org.elasticsearch.xcontent.ObjectParser.parseValue(ObjectParser.java:613)\n\t... 21 more\n"}
{"@timestamp":"2025-09-13T23:26:01.978Z", "log.level": "INFO", "message":"stopping ...", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch-shutdown","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elastic-tst-0","elasticsearch.cluster.name":"tst-cluster"}
{"@timestamp":"2025-09-13T23:26:01.995Z", "log.level": "INFO", "message":"stopped", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch-shutdown","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elastic-tst-0","elasticsearch.cluster.name":"tst-cluster"}
{"@timestamp":"2025-09-13T23:26:01.995Z", "log.level": "INFO", "message":"closing ...", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch-shutdown","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elastic-tst-0","elasticsearch.cluster.name":"tst-cluster"}
{"@timestamp":"2025-09-13T23:26:02.005Z", "log.level": "INFO", "message":"closed", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch-shutdown","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"elastic-tst-0","elasticsearch.cluster.name":"tst-cluster"}
{"@timestamp":"2025-09-13T23:26:02.008Z", "log.level": "INFO", "message":"Native controller process has stopped - no new native processes can be started", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"ml-cpp-log-tail-thread","log.logger":"org.elasticsearch.xpack.ml.process.NativeController","elasticsearch.node.name":"elastic-tst-0","elasticsearch.cluster.name":"tst-cluster"}
Basically I don’t know what to do with this issue, but founded:
Caused by: org.elasticsearch.xcontent.XContentParseException: [-1:22093] [data_stream] unknown field [failure_rollover_on_write]
I don’t know with what data stream it is related.