Overload

Hello,

Here is our configuration:

  • ES v1.7
  • 3 instances of ES running on 3 different server:

`free -m

                                   total       used       free     shared    buffers     cached

Mem: 7481 5813 1668 0 308 1744
-/+ buffers/cache: 3760 3721
Swap: 0 0 0`

  • cluster health:

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true' { "cluster_name" : "production", "status" : "green", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 210, "active_shards" : 395, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0 }

The issue is that we cannot insert anymore data in ES.

Each instance of ES are using more than 150% of CPU (here 194):
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 19424 elastic 20 0 5677m 2.8g 41m S 194 31.4 82:27.91 java 9 root 20 0 0 0 0 S 0 0.0 1782:53 rcuos/1/

Here is some logs:

[2019-07-22 13:19:42,224][DEBUG][action.search.type ] [pipeline01-production] [client_0][2], node[yC7Ul7VjTOSBEje9CMN8LQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@623212a8] org.elasticsearch.transport.NodeDisconnectedException: [pipeline00-production][inet[/10.240.210.211:9300]][indices:data/read/search[phase/query]] disconnected

[2019-07-22 13:20:10,474][DEBUG][action.search.type ] [pipeline01-production] [client_1][3], node[Qj9aBuwDTz25eshJUa2q9w], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@7631aed0] lastShard [true] org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 10000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@55b7138

Let me know if it miss informations,
Thanks in advance