Hello,
Here is our configuration:
- ES v1.7
- 3 instances of ES running on 3 different server:
`free -m
total used free shared buffers cached
Mem: 7481 5813 1668 0 308 1744
-/+ buffers/cache: 3760 3721
Swap: 0 0 0`
- cluster health:
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true' { "cluster_name" : "production", "status" : "green", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 210, "active_shards" : 395, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0 }
The issue is that we cannot insert anymore data in ES.
Each instance of ES are using more than 150% of CPU (here 194):
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 19424 elastic 20 0 5677m 2.8g 41m S 194 31.4 82:27.91 java 9 root 20 0 0 0 0 S 0 0.0 1782:53 rcuos/1/
Here is some logs:
[2019-07-22 13:19:42,224][DEBUG][action.search.type ] [pipeline01-production] [client_0][2], node[yC7Ul7VjTOSBEje9CMN8LQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@623212a8] org.elasticsearch.transport.NodeDisconnectedException: [pipeline00-production][inet[/10.240.210.211:9300]][indices:data/read/search[phase/query]] disconnected
[2019-07-22 13:20:10,474][DEBUG][action.search.type ] [pipeline01-production] [client_1][3], node[Qj9aBuwDTz25eshJUa2q9w], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@7631aed0] lastShard [true] org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 10000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@55b7138
Let me know if it miss informations,
Thanks in advance