We're using BulkProcessor for indexing and stumbled upon a problem when doing rolling restarts of our Elasticsearch cluster consisting of three nodes.
It appears that once the master node is going down, we get
ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/2/no master];]
and obviously fail to index bulk requests.
Two questions here:
- Does this need to be handled on the client side or is there any way to avoid this via cluster configuration?
- If we need to handle this in code, are there any best practices or suggested ways of handling failed bulks? We want to retry the failed data, but this would probably involve some not-trivial implementation to re-queue this data, delay or exponential backoff, combining retry data with newly arriving index requests, etc