# Detect failed data node

**URL:** <https://discuss.elastic.co/t/detect-failed-data-node/71410>\
**Category:** Elasticsearch\
**Created:** [January 12, 2017, 4:45pm UTC](https://discuss.elastic.co/t/detect-failed-data-node/71410 "2017-01-12T16:45:15Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![kay\_kay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kay_kay/32/1470_2.png) [@kay\_kay](https://discuss.elastic.co/u/kay_kay)\
**Post date:** [January 12, 2017, 4:45pm UTC](https://discuss.elastic.co/t/detect-failed-data-node/71410/1 "2017-01-12T16:45:15Z")

</div>

I have a setup with clients, masters and data nodes (elasticsearch 2.4.1).

When data node has memory problems (i.e. "Caused by: java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects") it tries to restore its state, but failed.

As a result I have a situation when there is a failed data node. Elasticsearch thinks that this data node still works, but it doesn't. Master node has the following logs:

`ReceiveTimeoutTransportException[[es-data-4][10.1.53.3:9300][cluster:monitor/nodes/stats[n]] request_id [1101574] timed out after [15000ms]]`

So it knows that this node is misfunctional.

But parsing master node is not a proper way to detect that. I'd like to monitor node state using restapi. And restapi shows that everything is fine. So is there a way to detect failed node?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 9, 2017, 4:46pm UTC](https://discuss.elastic.co/t/detect-failed-data-node/71410/2 "2017-02-09T16:46:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.