We use Elastic Cloud hosted on GCP, our production cluster is 1.41 TB storage, 32 GB RAM, 8 vCPU split in 2 zones.
We have a Java application which uses the ES as a database and yesterday, when we hit 20+ request per second, the cluster started rejecting our request to it with 502 Bad Gateway
org.elasticsearch.client.ResponseException: method [POST], host [https://<name>.es.us-central1.gcp.cloud.es.io:9243], URI [/<index>/_count], status line [HTTP/1.1 502 Bad Gateway]
{"ok":false,"message":"The instance rejected the connection."}
This happens for some of the requests, not all of them during that period and as far as I can see all other requests are going smootless, like 5-10ms response time.
I managed to reproduce it from my local machine as well and if I change my IP the 502 Bad Gateway is gone, so I think it may be some kind of IP throttling?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.