I am getting elasticsearch.exceptions.TransportError: TransportError(429, '429 Too Many Requests for my "_search" & "_bulk" operations.
My cluster health is green.
I have monitored threadpool write & search queues. Number for requests in queue never spike above 10 for both queues.
CPU utilization sometimes goes till 85%. JVM memory utilization stays in range 70-85%.
Which version of Elasticsearch are you using?
-> OpenSearch 1.1
What is the size, layout and configuration of your cluster?
-> This is our dev cluster with 1 data node only, EBS volume size- 15 GiB
What is the hardware specification?
-> t3.medium.search (vCPU-2, memory-4 GB)
How many indices and shards do you typically target in searches? How many concurrent searches are you running?
-> we are targeting 1 index only with 1 shards and running 10 concurrent searches maximum
How are you indexing into the cluster?
-> via RestHighLevelClient in Java.
How many indices and shards are you actively indexing into?
-> Regularly indexing into a single index (the one in which we search too) with 1 shard and creating new indices by filtering data from this source index.
How many concurrent indexing threads do you have?
-> concurrent indexing threads : 2 with queue size of 1000 for search and 10000 for write.
)