Does in-flight request breaker really track writing requests?

Elastic Stack Elasticsearch
1

According to the comment above CircuitBreaker#IN_FLIGHT_REQUESTS:

    /**
     * The in-flight request breaker tracks bytes allocated for reading and
     * writing requests on the network layer.
     */
    String IN_FLIGHT_REQUESTS = "inflight_requests";

The in-flight request breaker tracks both reading(receiving) requests and writing(sending) requests, but I only find it is used to track reading (receiving) requests in Netty4MessageChannelHandler#channelRead(transport level) and Netty4HttpPipeliningHandler#channelRead(http level).

2

I think you're right, this only tracks inbound requests.

1 Like
3

Moving your follow-up question here:

I am wondering if we should track writing requests by enhancing CircuitBreaker#IN_FLIGHT_REQUESTS or adding a new breaker.
Normally, it's ok to keep these writing requests untracked. But there may be some case which need many memory to send request, e.g. A coordinator send ShardSearchRequest to many nodes, or master send clusterstate to many nodes.

Eh maybe. We used to use the requests breaker for this but stopped. I'm not sure it really helped much. Indeed I wonder if the inflight_requests breaker even makes much sense any more.

In the two specific cases you called out, the outbound ShardSearchRequest overhead should be pretty trivial following Introduce batched query execution and data-node side reduce (#121885) by original-brownbear · Pull Request #126563 · elastic/elasticsearch · GitHub, and the cluster state the master sends out is already using shared buffers so should be manageable too. At least, there's not really much point in throwing a CircuitBreakingException on the master if it can't allocate a buffer for the outbound cluster state, because that will also cause an outage just as if the master went OOM. Instead we should do Limit cluster state transport message sizes · Issue #124068 · elastic/elasticsearch · GitHub.

1 Like