Master node heavy logging due to data node OS issues

Elastic Stack Elasticsearch
#1

Hello,

Due to some underlying OS issues, seeing this messages that are quickly feeling up the master node logs, causing issues with disk availability

Caused by: RemoteTransportException[[prod566][10.3.10.86:9300][internal:gateway/local/started_shards[n]]]; nested: ElasticsearchException[failed to load started shards]; nested: NotSerializableExceptionWrapper[/home/t/elasticsearch-2.1.2/data/herd-dev-ev1/nodes/0/indices/f5-2016.06.13/0/_state: Too many open files in system];

What's the best way to handle this problem? Do i force the node out of the cluster? Usually, the data node is unresponsive, so I can't ssh to it.

#2

You can change logging levels dynamically with this. You can probably use that to silence the log but it will hide any other errors.

Hard resetting the node might be the best thing if you think it is OS issues. It certainly looks like a busted node.

#3

My logging level at the moment is set to default: INFO
What I wanted to know more is to force that node to be taken out of rotation, but since I don't have ssh to the node at that moment to bring down elasticsearch, is there a way to dynamically issue this?

#4