Error unavailable_shards_exception when running password reset

[jrdtrstn]

I have shell scripts that automate the installation and configuration of ES 5.6. For security reasons, part of that bootstrap script is to change the default passwords for the system users (elastic, kibana, logstash). Since everything is automated, I have to verify that ES is running before I proceed with my commands to change the password. To do this, I have a loop that runs a curl command to _cat/health while waiting to get a response with a status of "green". Once I know the cluster has a green status, I then run commands to reset the passwords by running curl commands to _xpack/security/user/elastic/_password.

This process works most of the time, but there are instances where the password reset fails. Even though I get a green status from the curl status command:
Fri Feb 8 07:34:01 UTC 2019 [1] 1549611241 07:34:01 elastic-dev green 2 0 0 0 0 0 0 14 939.8ms 100.0%
...I still get the following error when running the password reset in some of my deployments:
Fri Feb 8 07:35:32 UTC 2019 [ERROR] {"error":{"root_cause":[{"type":"unavailable_shards_exception","reason":"[.security][0] [1] shardIt, [0] active : Timeout waiting for [1m], request: indices:data/write/update"}],"type":"unavailable_shards_exception","reason":"[.security][0] [1] shardIt, [0] active : Timeout waiting for [1m], request: indices:data/write/update"},"status":503}

Any thoughts on why I still get unavailable shards exception if my cluster was already green some milliseconds ago?

Are there more reliable ways to validate the cluster or shard status before running additional commands?

[Albert_Zaharovits]

Hi @jrdtrstn,

I can hypothesize that the cluster is green, but without the .security index. I need to see the cluster logs to be sure.

The pragmatic recommendation is to get back into the _cat/health poling if _xpack/security/user/elastic/_password fails. You can check the health of the .security index, and a cluster health of yellow is enough for write to succeed.