Cluster state often in yellow: data node and master are running but failed to ping each other

[MarkDai]

Found the reason, it's a bug of ES 5.1.1.
Hope anyone else can get there easily.

ES 5.1.1: Cluster loses a node randomly every few hours. Error: Message not fully read (response) for requestId

Hi all, I'm wondering if anybody else has seen this, or can help me understand what exactly is going on. We have a 5 data nodes/3 masters/1client node cluster at version 5.1.1. The data nodes are usually fairly busy, indexing around 5000 documents/s for primary shards. They seem to be keeping up with load though. Although every few hours (not always the same number of hours/times in the day) the cluster loses one of the nodes(not always the same) and as expected the replicas are promoted to…