Update taking more than 60 seconds.
WARNING:elasticsearch:POST http://elasticsearch.es:9200/profiles/doc/x_DeumMBCUnYUtZ/_update?refresh=false&version=9 [status:N/A request:60.059s]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@5246a6f2 on QueueResizingEsThreadPoolExecutor[name = es-data-0/search, queue capacity = 50, min queue capacity = 10, max queue capacity = 2000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 8.9ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@6165f54b[Running, pool size = 30, active threads = 1, queued tasks = 50, completed tasks = 279405]]
When there are 1k concurrent users trying to update their profiles and query their profiles.
I have 5 shards not crossing 3 GB per shard.
- 5 master nodes
- 6 data nodes(12 GB RAM / 6 GB Heap, 2 cores)
- SSD 30 GB
- 200k documents
Refresh Interval: 30s
thread_pool: search: size: 30 queue_size: 500 min_queue_size: 1000 max_queue_size: 5000 auto_queue_frame_size: 2000 target_response_time: 3s
Should I change/add any configuration? Is there something wrong with my setup? Any help would be appreciated.