Elasticsearch won’t start after upgrade to 8.4.1

Elastic Stack Elasticsearch
1

Similar issue, but this ticket has since been closed:

I upgraded Elasticsearch from 7.17.5 to 8.4.1, and unfortunately I did not check for any incompatibilities. I am now unable to start Elasticsearch, and downgrading is also not an option, since Elasticsearch prohibits me from doing so.

There is data that I would like to save and want to restore once ES is back up and running. Is there a way to do this?
Is there a way to just delete the affected index (status) that is preventing ES to start up and save the rest of my indices? Or is there a way to change the mapping when Elasticsearch is offline?

FYI I am running a single node cluster, so manipulating the file system might be easier than in a multi-cluster environment.

Thank you in advance for any pointers or help

Full stack trace:

"stacktrace": ["java.lang.IllegalStateException: Failed to parse mappings for index [[status/YEjHg9rJR5S6lBj9330JHA]]",
"at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkMappingsCompatibility(IndexMetadataVerifier.java:201) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.verifyIndexMetadata(IndexMetadataVerifier.java:93) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:225) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:212) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:126) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.node.Node.start(Node.java:1247) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.bootstrap.Elasticsearch.start(Elasticsearch.java:436) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:229) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67) ~[elasticsearch-8.4.1.jar:?]",
"Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed to parse mapping: Error parsing [format] on field [nextFetchDate]: Invalid format: [dateOptionalTime]: Unknown pattern letter: t",
"at org.elasticsearch.index.mapper.MapperService.parseMapping(MapperService.java:372) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:347) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:337) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkMappingsCompatibility(IndexMetadataVerifier.java:197) ~[elasticsearch-8.4.1.jar:?]",
"... 8 more",
"Caused by: java.lang.IllegalArgumentException: Error parsing [format] on field [nextFetchDate]: Invalid format: [dateOptionalTime]: Unknown pattern letter: t",
"at org.elasticsearch.index.mapper.DateFieldMapper$Builder.buildFormatter(DateFieldMapper.java:292) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:346) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:222) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.ObjectMapper$Builder.buildMappers(ObjectMapper.java:149) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.RootObjectMapper$Builder.build(RootObjectMapper.java:110) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:99) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:94) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.MapperService.parseMapping(MapperService.java:370) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:347) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:337) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkMappingsCompatibility(IndexMetadataVerifier.java:197) ~[elasticsearch-8.4.1.jar:?]",
"... 8 more",
"Caused by: java.lang.IllegalArgumentException: Invalid format: [dateOptionalTime]: Unknown pattern letter: t",
"at org.elasticsearch.common.time.DateFormatters.forPattern(DateFormatters.java:2129) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.common.time.DateFormatter.forPattern(DateFormatter.java:116) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.DateFieldMapper$Builder.buildFormatter(DateFieldMapper.java:286) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:346) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:222) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.ObjectMapper$Builder.buildMappers(ObjectMapper.java:149) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.RootObjectMapper$Builder.build(RootObjectMapper.java:110) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:99) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:94) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.MapperService.parseMapping(MapperService.java:370) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:347) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:337) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkMappingsCompatibility(IndexMetadataVerifier.java:197) ~[elasticsearch-8.4.1.jar:?]",
"... 8 more",
"Caused by: java.lang.IllegalArgumentException: Unknown pattern letter: t",
"at java.time.format.DateTimeFormatterBuilder.parsePattern(DateTimeFormatterBuilder.java:1891) ~[?:?]",
"at java.time.format.DateTimeFormatterBuilder.appendPattern(DateTimeFormatterBuilder.java:1781) ~[?:?]",
"at org.elasticsearch.common.time.DateFormatters.forPattern(DateFormatters.java:2126) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.common.time.DateFormatter.forPattern(DateFormatter.java:116) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.DateFieldMapper$Builder.buildFormatter(DateFieldMapper.java:286) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:346) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.DateFieldMapper$Builder.build(DateFieldMapper.java:222) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.ObjectMapper$Builder.buildMappers(ObjectMapper.java:149) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.RootObjectMapper$Builder.build(RootObjectMapper.java:110) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:99) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:94) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.MapperService.parseMapping(MapperService.java:370) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:347) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:337) ~[elasticsearch-8.4.1.jar:?]",
"at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkMappingsCompatibility(IndexMetadataVerifier.java:197) ~[elasticsearch-8.4.1.jar:?]",
"... 8 more"] }
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/my-cluster.log
{"type": "server", "timestamp": "2022-09-16T14:23:36,664Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "4bb22c5e1dee", "message": "stopping ..." }
{"type": "server", "timestamp": "2022-09-16T14:23:36,712Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "4bb22c5e1dee", "message": "stopped" }
{"type": "server", "timestamp": "2022-09-16T14:23:36,712Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "4bb22c5e1dee", "message": "closing ..." }
{"type": "server", "timestamp": "2022-09-16T14:23:36,722Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "4bb22c5e1dee", "message": "closed" }
{"type": "server", "timestamp": "2022-09-16T14:23:36,725Z", "level": "INFO", "component": "o.e.x.m.p.NativeController", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "4bb22c5e1dee", "message": "Native controller process has stopped - no new native processes can be started" }

ERROR: Elasticsearch exited unexpectedly
2

This is an issue that seems to be fixed for the next version, 8.5.

But it probably will take a time until this version is released.

You do not have many options, manipulating the file system is always a risk and can lead to data loss, but you can do nothing with the service offline.

Have you tried to downgrade? I think that in this case Elasticsearch is failing in a point before it migrates your indices to a next version and a downgrade may work, but I'm not sure, in any case, copy the entire data directory to have at least a file system backup.

You can also try to manually delete the index folder to see if you can at least start the system, make a snapshot of your other indices and then reinstall it correctly.

Another option would be to build the main branch from github, which seems to have this fix applied according to the previous commit.

3

Thank you for the prompt response, Leandro.
Unfortunately, downgrading has not worked. I have tried every version from ES 8.4.1 down to and including 7.17.5. Every v8 fails with the MapperParsingException, and v7.17.5 fails with an error message that the index has been created by ES 8.x.x
I will try manually deleting the index giving me problems and will let you know if that works. Otherwise, I will have to wait for v8.5.

4

If you can wait for the fix in 8.5, then I recommend that you wait for it instead of manually changing anything in the file system.

5

Yes, it looks like I will have to wait. I have deleted the directory that is mentioned in the stacktrace under

/usr/share/elasticsearch/data/YEjHg9rJR5S6lBj9330JHA

but I am stiil getting the same error in my first post. It looks like there is still some reference to this index elsewhere

grep -iR "YEjHg9rJR5S6lBj9330JHA" *
grep: _state/_205.cfs: binary file matches

I am definitely not touching these binary files, and I'm glad that I made a backup before deleting the aforementioned directory.

Thank you for your help!

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.