Elasticsearch node won't start after updating index setting with index.mapper.dynamic

Elastic Stack Elasticsearch
1

Hello,

I have an Elasticsearch cluster with version 7.17.2. I updated one of my indices with the following setting:

PUT /testindex/_settings { "index.mapper.dynamic": true }
and everything was OK, my settings were updated successfully.

After some time, i restarted the elasticsearch service on one of my nodes but the service doesn't start. In the logs i can see this error:

[2024-01-15T17:53:03,766][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [pr-esaas-fe-p192c254md-01] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Failed to parse mappings for index [[testindex/Qj8OReW1QSGEKucNcYwCrw]]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.17.2.jar:7.17.2]
	at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.17.2.jar:7.17.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.17.2.jar:7.17.2]
Caused by: java.lang.IllegalStateException: Failed to parse mappings for index [[testindex/Qj8OReW1QSGEKucNcYwCrw]]
	at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkMappingsCompatibility(IndexMetadataVerifier.java:191) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.verifyIndexMetadata(IndexMetadataVerifier.java:87) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:279) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:265) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:178) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.node.Node.start(Node.java:1188) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:335) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:443) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.17.2.jar:7.17.2]
	... 6 more
Caused by: java.lang.IllegalArgumentException: Setting index.mapper.dynamic was removed after version 6.0.0
	at org.elasticsearch.index.mapper.MapperService.<init>(MapperService.java:208) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkMappingsCompatibility(IndexMetadataVerifier.java:177) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.verifyIndexMetadata(IndexMetadataVerifier.java:87) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:279) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:265) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:178) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.node.Node.start(Node.java:1188) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:335) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:443) ~[elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.17.2.jar:7.17.2]
	... 6 more

Every action performed on this index will result in an error stating that "Setting index.mapper.dynamic was removed after version 6.0.0".
So, i am able to successfully add this setting, but i am not able to remove it or start the service on this node again!!
I also deleted the index from the cluster, still no success in starting elasticsearch service on this node. It seems there is no way to recover it.
what else can i do? is a production cluster, so i am not able to easily upgrade/perform important changes. Please help!

2

Unfortunately I'm not sure how you can recover from this state.

Being able to add this setting on 7.17.X is a bug that is tracked in this github issue, but there is no fix for it yet.

@DavidTurner sorry to tag you, but is there any way to recove the node in this case?

3

I don't think so unfortunately, but as long as it's just one node (and the cluster health is not red) then you can delete the data path contents and it'll start as a new fresh node and rejoin the cluster.

4

Hello!

Thanks for the fast reply!
I managed to recreate the node from scratch and rejoin the cluster as a new node.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.