Today,one of my data node's cpu usage suddenly rise to 100%(I have 10 datanodes,ony one node's cpu rise).At that time,all api has no response and no timeout (bulk api,stats api and so on).But i can ping the datanode.
For example,the api curl 'localhost:9200/_cat/thread_pool?v&h=id,host,bulk.size,bulk.active,bulk.rejected,bulk.completed,bulk.queueSize,bulk.queue'. has no response.
Here are master node's log:
NodeDisconnectedException[[es165-4][ip:port][cluster:monitor/nodes/info[n]] disconnected
NodeDisconnectedException[[es165-4][ip:port][cluster:monitor/nodes/info[n]] disconnected
NodeDisconnectedException[[es165-4][ip:port][cluster:monitor/nodes/info[n]] disconnected
The died state continue to i shutdown the datanode host.
es version:V2.3.3