ElasticSearch Timeout Error during heavy write load

Elastic Stack Elasticsearch
1

Error:
Update taking more than 60 seconds.

WARNING:elasticsearch:POST http://elasticsearch.es:9200/profiles/doc/x_DeumMBCUnYUtZ/_update?refresh=false&version=9 [status:N/A request:60.059s]

Also getting

Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@5246a6f2 on QueueResizingEsThreadPoolExecutor[name = es-data-0/search, queue capacity = 50, min queue capacity = 10, max queue capacity = 2000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 8.9ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@6165f54b[Running, pool size = 30, active threads = 1, queued tasks = 50, completed tasks = 279405]]

When:

When there are 1k concurrent users trying to update their profiles and query their profiles.
I have 5 shards not crossing 3 GB per shard.

Resources
Kubernetes

  • 5 master nodes
  • 6 data nodes(12 GB RAM / 6 GB Heap, 2 cores)
  • SSD 30 GB
  • 200k documents

Config:

Refresh Interval: 30s

thread_pool:
  search:
    size: 30
    queue_size: 500
    min_queue_size: 1000
    max_queue_size: 5000
    auto_queue_frame_size: 2000
    target_response_time: 3s

Should I change/add any configuration? Is there something wrong with my setup? Any help would be appreciated.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.