Improving resiliency or changing settings of system indices

[garethhumphriesgkc (Gareth)]

Hi,

I'm trying to improve the resiliency of my elastic stack. I want to make it tolerant to any two node failures, but I can't update system indices to have more than one replica.

The setting in question is index.auto_expand_replicas - I'd like to make this for example 0-2 instead of 0-1, but when I try I get:

PUT /.kibana_7.17.2_001/_settings
{"index.auto_expand_replicas":"0-2"}

{
  "type":"security_exception",
  "reason":"action [indices:admin/settings/update] is unauthorized for user [<user>] with effective roles [kibana_admin,kibana_monitoring,modify_system_indices,monitoring_user,superuser] on restricted indices [.kibana_7.17.2_001], this action is granted by the index privileges [manage,all]"
}

As you can see, I've created a role to grant all to .* in an attempt to explicitly allow it, but still no dice.

Is there any way I can modify these settings?

There are some other settings I'd like to be able to change too, so I can control the addition of new data nodes to my cluster - right now I can't control when system indices get allocated to a new node, so I have to just hope everything goes well rather than cut indices across slowly.

Any tips on how I can change settings on system indices much appreciated.

[garethhumphriesgkc (Gareth)]

Seems no-one else has any ideas either. Does that mean that, fundamentally, Elasticsearch just doesn't support any better than N+1 resiliency?

[warkolm (Mark Walkom)]

AFAIK we don't support changing things for system indices as they will be overwritten the next time you upgrade.

That's probably not what you want to hear sorry, I would suggest raising a feature request on GitHub to see what can be done.

[garethhumphriesgkc (Gareth)]

Not ideal, but nice to get confirmation I'm not missing something.

Sure thing, I'll raise an FR. Thanks for responding.