We are using elasticsearch 6.7.2 deployed on k8s with a configuration of 3 master, 4 data, 2 client nodes.
When setting some ILM Policies, I am seeing that it is immediately returning with "acknowledged": false. My understanding is that this should only happen if there is a timeout on one of the nodes, and that the acknowledged: false is not necessarily an error, but just indicates that the operation could not complete before the timeout.
I am a bit confused by this as I thought the default timeout was either 10s or 30s, so the fact that it returns false immediately is confusing to me. My code relies on this operation being completed before trying to create any indices to make sure the policy is successfully updated. When i was running this with a different configuration (3 master, 2 data, 2 client), it would always return acknowledged as true.
My question is, why would this be returning acknowledged: false so quickly, and how can I make this operation wait for acknowledged: true before returning.