# Elasticsearch 6.3.0 doesn't retry on index replica bulk write failure
**URL:** https://discuss.elastic.co/t/elasticsearch-6-3-0-doesnt-retry-on-index-replica-bulk-write-failure/174305
**Category:** Elasticsearch
**Created:** [March 28, 2019, 10:31am UTC](https://discuss.elastic.co/t/elasticsearch-6-3-0-doesnt-retry-on-index-replica-bulk-write-failure/174305 "2019-03-28T10:31:00Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@andrejbl](https://discuss.elastic.co/u/andrejbl)
#### Post date: [March 28, 2019, 10:31am UTC](https://discuss.elastic.co/t/elasticsearch-6-3-0-doesnt-retry-on-index-replica-bulk-write-failure/174305/1 "2019-03-28T10:31:00Z")
(original discussion on github - [https://github.com/elastic/elasticsearch/issues/39259](https://github.com/elastic/elasticsearch/issues/39259))
We are running a number of large-scale Elasticsearch clusters (40+ nodes, TBs of data, ingestion and search rates in 100-1000s per second) on ES version 6.3.0. When doing rolling upgrades of the individual nodes (that include a machine reboot) we have observed brief network disconnects of unrelated nodes. E.g. if node1 is being rebooted, ES on node25 would throw socket closed exceptions unrelated to connections to node1. As a direct impact, a number of bulk indexing operations on node25 would fail on node's shard replicas and master would mark the corresponding shards as stale:
```auto
failing shard [failed shard, shard [redacted][5], node[GLG7S_x_Sa25ey3vjmX_PA], [R], s[STARTED], a[id=3jAVAHveT1eAsf75zN4hmw], message [failed to perform indices:data/write/bulk[s] on replica [redacted][5]
```
```auto
2019-02-22T12:32:38,022][WARN][o.e.c.r.a.AllocationService] [es-master-3.localdomain] failing shard [failed shard, shard [redacted][5], node[GLG7S_x_Sa25ey3vjmX_PA], [R], s[STARTED], a[id=3jAVAHveT1eAsf75zN4hmw], message [failed to perform indices:data/write/bulk[s] on replica [redacted][5], node[GLG7S_x_Sa25ey3vjmX_PA], [R], s[STARTED], a[id=3jAVAHveT1eAsf75zN4hmw]], failure [NodeNotConnectedException[[node25.localdomain][redacted:9300] Node not connected]], markAsStale [true]]
```
As discussed on the github issue, we do not see any retries of those replica bulk indexing write operations. It seems as if the node owning the primary shard will try to write to replica once and then give up. The end result is a loss of a number of replicas, lots of shard rebalancing and prolonged yellow cluster state.
Is there a functional reason for the bulk replica writes not to be more robust? Giving up after a single retry seems rather flakey.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [March 28, 2019, 11:13am UTC](https://discuss.elastic.co/t/elasticsearch-6-3-0-doesnt-retry-on-index-replica-bulk-write-failure/174305/2 "2019-03-28T11:13:28Z")
Hi @andrejbl, thanks for continuing the discussion here.
> [@andrejbl](#):
>
> Is there a functional reason for the bulk replica writes not to be more robust? Giving up after a single retry seems rather flakey.
A network disconnection occurs under the following situations:
- the connection is dropped at the OS level after some number of unacknowledged retransmissions (15 by default on Linux)
- the remote node repeatedly failed to respond to application-level health checks (3 times by default, and only on connections to or from the master node)
- the connection was actively terminated by the remote node, for instance because it is shutting down
In all cases it makes sense to treat a network disconnection as a fatal error and not to retry further. The alternative is to queue up requests for the remote node in the hope that it reconnects soon, but this is normally not a helpful thing to do.
> [@andrejbl](#):
>
> When doing rolling upgrades of the individual nodes (that include a machine reboot) we have observed brief network disconnects of unrelated nodes. E.g. if node1 is being rebooted, ES on node25 would throw socket closed exceptions unrelated to connections to node1.
This is unexpected and warrants further investigation, as I think this sits at the heart of the problem you were describing on GitHub. Can you share the logs surrounding this event? I'd like to see the logs from:
- the elected master node
- the rebooting node (`node1` above)
- the node with the exceptions (`node25` above)
- the peers from which it disconnected.
They're probably quite large so perhaps use [https://gist.github.com/](https://gist.github.com/). It would also be good to see the output of `GET _nodes/stats` to help correlate node IDs/IP addresses/etc.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 25, 2019, 11:13am UTC](https://discuss.elastic.co/t/elasticsearch-6-3-0-doesnt-retry-on-index-replica-bulk-write-failure/174305/3 "2019-04-25T11:13:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.