Elastic node can't start due to removed index setting

[vvp199303]

Hi. I have a trouble with Elastic cluster (v 7.13.0) . Аfter adding an { "index.mapper.dynamic" : false } option for one index in cluster I was unable to restart cluster node.
Also, I can't change "index.mapper.dynamic" option to true or null - I get similar error:

curl -X PUT "http://localhost:9200/core_ws_folders_documents/_settings?pretty" -H 'Content-Type: application/json' -d' {  "index.mapper.dynamic" : true} '{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "Setting index.mapper.dynamic was removed after version 6.0.0"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "Setting index.mapper.dynamic was removed after version 6.0.0"
  },
  "status" : 400
}

More information about the problem in attached file.

I don't quite understand how you can get around this problem. Any help would be appreciated. Thanks.

[warkolm (Mark Walkom)]

Welcome to our community!

Please don't post pictures of text, logs or code. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them

[DineshNaik (Dinesh Kumar Naik)]

As you see in logs as well as api output : Setting index.mapper.dynamic was removed after version 6.0.0

Ideally, it should not impact the es process per say. Are you saying es process/jvm is not running once you ran the api to change it?

i guess only the index for which you tried to change the setting might be in red status and probably you need to recreate the index and repopulate the data. See if you can delete the index which is in red status if that's an option for you?

Also avoid changing index.mapper.dynamic going forward as its not recommended!

[vvp199303]

@DineshNaik

Ideally, it should not impact the es process per say. Are you saying es process/jvm is not running once you ran the api to change it?

Yes, a problem appeared after I changed setting to false. Index is still yellow due to replica shards.
If there is no another solution I will try to recreate index, but that's not a good option for me.
Index size is more than 4 TB, reindexing will take a long time.
Also, this is current index settings:

admin@elastic2:~$ curl -s -X GET  "http://localhost:9200/core_ws_folders_documents/_settings?pretty"
{
  "core_ws_folders_documents" : {
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "mapping" : {
          "total_fields" : {
            "limit" : "50000"
          }
        },
        "number_of_shards" : "8",
        "provided_name" : "core_ws_folders_documents",
        "mapper" : {
          "dynamic" : "false"
        },
        "creation_date" : "1625641062374",
        "number_of_replicas" : "1",
        "uuid" : "WnQR8vlxT_iu4XsCY99cuw",
        "version" : {
          "created" : "7130099"
        }
      }
    }
  }
}

[DineshNaik (Dinesh Kumar Naik)]

Ok , I think it's a bug . In ideal scenario we should not be able to modify it at the first place as that caused the damage and index went in yellow/red status.

By the way, once you run the api, did you restart es process and saw the indices in yellow status or without restart ?

[DineshNaik (Dinesh Kumar Naik)]

@vvp199303 I am not sure how to fix this on the existing index, but I would suggest you to either take a backup of your index or make the cluster status green as soon as possible before you restart your cluster(all nodes).

In fact, i would recommend not to restart the nodes unless you are done fixing this issue, restarting nodes will make the index go in red status and you will loose the data.

Here is one way to fix it :

1. create a new index with just primary shard/shards (no replica, by making replca 1) . say core_ws_folders_documents_new

2. reindex data to this new index core_ws_folders_documents_new using reindex api
   you can use command

POST _reindex?wait_for_completion=false
{
  "source": {
    "index": "core_ws_folders_documents"
  },
  "dest": {
    "index": "core_ws_folders_documents_new"
  }
}

the above command will give you a task id through which you can check the status of reindex api:

GET _tasks/TASK_ID

3. once reindexing is done you can increase the number of replica for the newly created index.

4. Delete the index core_ws_folders_documents

DELETE core_ws_folders_documents

5. create an alias for index core_ws_folders_documents_new as core_ws_folders_documents

POST _aliases
{
  "actions": [
    {
      "add": {
        "index": "core_ws_folders_documents_new",
        "alias": "core_ws_folders_documents"
      }
    }
  ]
}

Note: Please do try these in a lower environment first before you actually run them in your actual cluster.