# Could we support timeout mechanism for replica bulk request？
**URL:** https://discuss.elastic.co/t/could-we-support-timeout-mechanism-for-replica-bulk-request/316977
**Category:** Elasticsearch
**Created:** [October 19, 2022, 9:31am UTC](https://discuss.elastic.co/t/could-we-support-timeout-mechanism-for-replica-bulk-request/316977 "2022-10-19T09:31:48Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@howardhuang](https://discuss.elastic.co/u/howardhuang)
#### Post date: [October 19, 2022, 9:31am UTC](https://discuss.elastic.co/t/could-we-support-timeout-mechanism-for-replica-bulk-request/316977/1 "2022-10-19T09:31:48Z")
Assume that shard has 1 primary + 1 replica, if a bulk request goes to primary and finished the shard bulk write operation, then transfer the bulk request to replica node, if the replica node is a really slow node, for example may get stuck for several minutes (cpu or memory issue, node network ping is ok, node cannot be removed from cluster), then all the bulk operation requests would get stuck before the slow replica node done.
In a 100+ nodes cluster, each node has the same index shard, if we have a single slow node, the above case may slow down the whole cluster bulk operations.
Could we add timeout mechanism for the replica bulk request? For example, if a replica got timeout after like 30s, then make the shard failed, don't block primary shard bulk operation forever.
I have done some test, I modified code to sleep 10mins in replica write operation, we could find that curl request took 10mins+:
```auto
/_bulk?pretty" -H 'Content-Type: application/json' -d'
> { "index" : { "_index" : "replica_test", "_id" : "1" } }
> { "field1" : "value1" }
> { "index" : { "_index" : "replica_test", "_id" : "2" } }
> { "field1" : "value2" }
> { "index" : { "_index" : "replica_test", "_id" : "3" } }
> { "field1" : "value3" }
> '
{
"took" : 600051,
"errors" : false,
"items" : [
{
"index" : {
"_index" : "replica_test",
"_type" : "_doc",
"_id" : "1",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"_seq_no" : 0,
"_primary_term" : 2,
"status" : 201
}
},
```
Related issue: [Support timeout mechanism for replica bulk request. · Issue #90981 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/90981)
---
### Author:  [@howardhuang](https://discuss.elastic.co/u/howardhuang)
#### Post date: [October 23, 2022, 2:48am UTC](https://discuss.elastic.co/t/could-we-support-timeout-mechanism-for-replica-bulk-request/316977/2 "2022-10-23T02:48:45Z")
Anyone could help?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [October 24, 2022, 10:04am UTC](https://discuss.elastic.co/t/could-we-support-timeout-mechanism-for-replica-bulk-request/316977/3 "2022-10-24T10:04:46Z")
You _could_ have a timeout here but it doesn't really make sense. Timing out individual requests to a bad node is a very weak response. If the node is as unhealthy as you describe, it typically should be removed from the cluster entirely.
Therefore I think a better approach would be to improve the node-level health checks that remove unhealthy nodes from the cluster.
---
### Author:  [@howardhuang](https://discuss.elastic.co/u/howardhuang)
#### Post date: [October 25, 2022, 3:09am UTC](https://discuss.elastic.co/t/could-we-support-timeout-mechanism-for-replica-bulk-request/316977/4 "2022-10-25T03:09:27Z")
@DavidTurner Thanks for your reply. I agree with you that we need to improve node-level health checks. Currently we remove nodes only if they could not PING success, that's based on network and it's only a node heartbeat. Shall we need to add more async unhealth check mechanisms?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [October 25, 2022, 7:50am UTC](https://discuss.elastic.co/t/could-we-support-timeout-mechanism-for-replica-bulk-request/316977/5 "2022-10-25T07:50:28Z")
> [@howardhuang](#):
>
> Currently we remove nodes only if they could not PING success, that's based on network and it's only a node heartbeat.
Today's checks also have a timeout, and they also ensure that the data path is writeable. But yes, one option would be to add more mechanisms within Elasticsearch here. There are some things that we can't really detect from within Elasticsearch (e.g. they need `root` or other privileges that Elasticsearch doesn't have, for instance looking at disk health via SMART metrics). Checks like that need to be built into your operating platform instead.
---
### Author:  [@howardhuang](https://discuss.elastic.co/u/howardhuang)
#### Post date: [October 25, 2022, 7:54am UTC](https://discuss.elastic.co/t/could-we-support-timeout-mechanism-for-replica-bulk-request/316977/6 "2022-10-25T07:54:40Z")
> [@DavidTurner](#):
>
> Today's checks also have a timeout, and they also ensure that the data path is writeable.
Thank you David. Could you please help to indicate which part of code on the github implements these checks?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [October 26, 2022, 1:39am UTC](https://discuss.elastic.co/t/could-we-support-timeout-mechanism-for-replica-bulk-request/316977/7 "2022-10-26T01:39:25Z")
The timeout on follower checks is applied here:
>
The writability check is here:
>
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 23, 2022, 1:40am UTC](https://discuss.elastic.co/t/could-we-support-timeout-mechanism-for-replica-bulk-request/316977/8 "2022-11-23T01:40:07Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.