Rally benchmarking Elasticsearch with 429 error code

Hello,

We are testing a new Elasticsearch cluster with Rally. The indexing task is getting an error rate greater than zero and we also found that the total number of indexed documents does not match the target number of the track.

In the "rally.log" file, we found a 429 code response on some requests:

2021-06-23 15:00:38,463 -not-actor-/PID:91 elasticsearch WARNING POST http://elasticsearch-data:9200/_bulk [status:429 request:0.241s]

However, Elasticsearch node logs does not register rejected requests.
According to other posts, Elasticsearch logs "rejected execution" when error 429 occurs, is it true?

Our Elasticsearch cluster is deployed into a proxied and balanced installation, and we suspect the 429 error could be coming from these components.

Thanks in advance.

In that case I would recommend running the same benchmark directly against the cluster and compare results.