# Master node heavy logging due to data node OS issues

**URL:** <https://discuss.elastic.co/t/master-node-heavy-logging-due-to-data-node-os-issues/52779>\
**Category:** Elasticsearch\
**Created:** [June 14, 2016, 7:48pm UTC](https://discuss.elastic.co/t/master-node-heavy-logging-due-to-data-node-os-issues/52779 "2016-06-14T19:48:06Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino)\
**Post date:** [June 14, 2016, 7:48pm UTC](https://discuss.elastic.co/t/master-node-heavy-logging-due-to-data-node-os-issues/52779/1 "2016-06-14T19:48:06Z")

</div>

Hello,

Due to some underlying OS issues, seeing this messages that are quickly feeling up the master node logs, causing issues with disk availability

Caused by: RemoteTransportException[[prod566][10.3.10.86:9300][internal:gateway/local/started\_shards[n]]]; nested: ElasticsearchException[failed to load started shards]; nested: NotSerializableExceptionWrapper[/home/t/elasticsearch-2.1.2/data/herd-dev-ev1/nodes/0/indices/f5-2016.06.13/0/\_state: Too many open files in system];

What's the best way to handle this problem? Do i force the node out of the cluster? Usually, the data node is unresponsive, so I can't ssh to it.

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [June 14, 2016, 8:04pm UTC](https://discuss.elastic.co/t/master-node-heavy-logging-due-to-data-node-os-issues/52779/2 "2016-06-14T20:04:44Z")

</div>

You can change logging levels dynamically with [this](https://www.elastic.co/guide/en/elasticsearch/guide/current/logging.html). You can probably use that to silence the log but it will hide any other errors.

Hard resetting the node might be the best thing if you think it is OS issues. It certainly looks like a busted node.

---

<div class="post-metadata">

**Author:** ![stecino](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@stecino](https://discuss.elastic.co/u/stecino)\
**Post date:** [June 14, 2016, 9:48pm UTC](https://discuss.elastic.co/t/master-node-heavy-logging-due-to-data-node-os-issues/52779/3 "2016-06-14T21:48:43Z")

</div>

My logging level at the moment is set to default: INFO  
What I wanted to know more is to force that node to be taken out of rotation, but since I don't have ssh to the node at that moment to bring down elasticsearch, is there a way to dynamically issue this?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:43pm UTC](https://discuss.elastic.co/t/master-node-heavy-logging-due-to-data-node-os-issues/52779/4 "2017-07-05T22:43:51Z")

</div>