Error:
Update taking more than 60 seconds.
WARNING:elasticsearch:POST http://elasticsearch.es:9200/profiles/doc/x_DeumMBCUnYUtZ/_update?refresh=false&version=9 [status:N/A request:60.059s]
Also getting
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@5246a6f2 on QueueResizingEsThreadPoolExecutor[name = es-data-0/search, queue capacity = 50, min queue capacity = 10, max queue capacity = 2000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 8.9ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@6165f54b[Running, pool size = 30, active threads = 1, queued tasks = 50, completed tasks = 279405]]
When:
When there are 1k concurrent users trying to update their profiles and query their profiles.
I have 5 shards not crossing 3 GB per shard.
Resources
Kubernetes
- 5 master nodes
- 6 data nodes(12 GB RAM / 6 GB Heap, 2 cores)
- SSD 30 GB
- 200k documents
Config:
Refresh Interval: 30s
thread_pool:
search:
size: 30
queue_size: 500
min_queue_size: 1000
max_queue_size: 5000
auto_queue_frame_size: 2000
target_response_time: 3s
Should I change/add any configuration? Is there something wrong with my setup? Any help would be appreciated.