The instance rejected the connection

[Ventsislav_Kyorovski]

Hello everyone,

We use Elastic Cloud hosted on GCP, our production cluster is 1.41 TB storage, 32 GB RAM, 8 vCPU split in 2 zones.

We have a Java application which uses the ES as a database and yesterday, when we hit 20+ request per second, the cluster started rejecting our request to it with 502 Bad Gateway

org.elasticsearch.client.ResponseException: method [POST], host [https://<name>.es.us-central1.gcp.cloud.es.io:9243], URI [/<index>/_count], status line [HTTP/1.1 502 Bad Gateway]
{"ok":false,"message":"The instance rejected the connection."}

This happens for some of the requests, not all of them during that period and as far as I can see all other requests are going smootless, like 5-10ms response time.

I managed to reproduce it from my local machine as well and if I change my IP the 502 Bad Gateway is gone, so I think it may be some kind of IP throttling?

Does anyone has any suggestions on this?

Thank you

[warkolm (Mark Walkom)]

Welcome to our community!

Yes there is rate limiting, Rate limiting | Elasticsearch Service Documentation | Elastic goes into that.

[Ventsislav_Kyorovski]

Thank you for the fast answer @warkolm, but isn't those limitations for the Elasticsearch Service API

A RESTful API is available with Elasticsearch Service, allowing you to perform most of the operations available in the UI console through API calls

Currently, when I call the Elasticsearch API it doesn't return the headers shown on the rate limit section

curl --location --request GET 'https://<cluster>.es.us-central1.gcp.cloud.es.io:9243/_cat/indices?v' \
--header 'Authorization: Basic <auth>' \
--data-raw ''

and one example response if that helps

< HTTP/2 200 
< content-type: text/plain; charset=UTF-8
< x-cloud-request-id: OBeXZnfqQLSJPVZX7VYNBw
< x-elastic-product: Elasticsearch
< x-found-handling-cluster: f3a79221db9f446fa0f40e17af27a1f2
< x-found-handling-instance: instance-0000000005
< content-length: 1096
< date: Tue, 21 Jun 2022 07:07:42 GMT
< 
health status index                                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   metrics-endpoint.metadata_current_default 2nHVKcRcQOSeEhtlHXBv5Q   1   1          0            0       450b           225b
green  open   companies                                 1_nnZb28TfCAMiQf2iZ8tw   1   1      30794            2     38.2mb         19.1mb
green  open   events                                    okU4LOwGS0iYhqKfjiXMvg   1   0    1729497            0     10.1gb         10.1gb
green  open   postings                                  28iJCXAfRzqkP9jEaS9ewQ   1   1       9710         3431       19mb         10.7mb
green  open   verified-companies                        aCnpyLF9TWiDpEIEVNMMzg   1   1       7845          240    966.7kb        472.1kb
green  open   .lists-default-000001                     ec7xvxk4StyObA4vkq-TWg   1   1          0            0       450b           225b
green  open   .items-default-000001                     X8BpGLTzTGW_RMUrtKnUyA   1   1          0            0       450b           225b