Add resiliency on .security-7 index

[Josselin (Mariette)]

Hi,

During multiple incident with cluster restart we lost the nodes where the index .security-7 was stored. It had a huge impact and we want to avoid as much as possible this situation to occur again.

We have seen on the index the setting : index.auto_expand_replicas which we would want to edit and tune to our needs.
We tried to update the settings using informations found on old thread / internet with no result.

In our last attempts we had a role with rights like :

{
  "indices": [
    {
      "names": ["*",".*"],
      "privileges": ["manage", "all"],
      "allow_restricted_indices": true
    }
  ]
}

but we got this issue when trying to update the settings :

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_state_exception",
        "reason": "Cannot override settings on system indices: [.security-[0-9]+*] -> [index.auto_expand_replicas]"
      }
    ],
    "type": "illegal_state_exception",
    "reason": "Cannot override settings on system indices: [.security-[0-9]+*] -> [index.auto_expand_replicas]"
  },
  "status": 500
}

If someone has information to share about this topic we would be glad

Thank you !

[vincenbr (Vincent Bréhin)]

Changing number of replica has been forbidden on system indices (including .security) . This might change in the future, you can refer to Productize a way to change index.auto_expand_replicas and index.number_of_replicas settings on the .security index · Issue #92992 · elastic/elasticsearch · GitHub

Tip: for cluster outages due to .security index being unavailable, you can set a "file realm" in order to mitigate the situation. See File-based user authentication | Elasticsearch Guide [8.8] | Elastic

[Josselin (Mariette)]

Thank you very much Vincent for your time and answer.

We will monitor the issue in hope this change in the future

We already have a file realm for some accounts but would expect all dynamic accounts not using a file realm to keep working