Due to some underlying OS issues, seeing this messages that are quickly feeling up the master node logs, causing issues with disk availability
Caused by: RemoteTransportException[[prod566][10.3.10.86:9300][internal:gateway/local/started_shards[n]]]; nested: ElasticsearchException[failed to load started shards]; nested: NotSerializableExceptionWrapper[/home/t/elasticsearch-2.1.2/data/herd-dev-ev1/nodes/0/indices/f5-2016.06.13/0/_state: Too many open files in system];
What's the best way to handle this problem? Do i force the node out of the cluster? Usually, the data node is unresponsive, so I can't ssh to it.