# 3 Node Elasticsearch cluster is failing repeatedly with error: this node is unhealthy: health check failed due to broken node lock **URL:** \ **Category:** Elasticsearch\ **Created:** [May 11, 2023, 5:42pm UTC](https://discuss.elastic.co/t/3-node-elasticsearch-cluster-is-failing-repeatedly-with-error-this-node-is-unhealthy-health-check-failed-due-to-broken-node-lock/333234 "2023-05-11T17:42:31Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![akansha.agarwal1](https://avatars.discourse-cdn.com/v4/letter/a/858c86/32.png) [@akansha.agarwal1](https://discuss.elastic.co/u/akansha.agarwal1)\ **Post date:** [May 11, 2023, 5:42pm UTC](https://discuss.elastic.co/t/3-node-elasticsearch-cluster-is-failing-repeatedly-with-error-this-node-is-unhealthy-health-check-failed-due-to-broken-node-lock/333234/1 "2023-05-11T17:42:31Z")
Hi All, I am stuck in a very weird situation. My 3-node ES cluster is failing after 8-10 days abruptly with error: [WARN][o.e.c.c.ClusterFormationFailureHelper] [elasticsearch-0.es-service] this node is unhealthy: health check failed due to broken node lock No change is being done during that duration. Neither any other instance of ES is being run ES Version: 8.6.2 Deployment: 3-node cluster on k8s Can anyone please suggest any solution or investigation points? Pleas Help! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 11, 2023, 5:47pm UTC](https://discuss.elastic.co/t/3-node-elasticsearch-cluster-is-failing-repeatedly-with-error-this-node-is-unhealthy-health-check-failed-due-to-broken-node-lock/333234/2 "2023-05-11T17:47:10Z")
That means that Elasticsearch saw a change in its data directory for which it was not responsible, so it stops all write activity to protect your data. To fix it, remove any other process that might make such a change and then restart Elasticsearch. ---
**Author:** ![akansha.agarwal1](https://avatars.discourse-cdn.com/v4/letter/a/858c86/32.png) [@akansha.agarwal1](https://discuss.elastic.co/u/akansha.agarwal1)\ **Post date:** [May 11, 2023, 5:57pm UTC](https://discuss.elastic.co/t/3-node-elasticsearch-cluster-is-failing-repeatedly-with-error-this-node-is-unhealthy-health-check-failed-due-to-broken-node-lock/333234/3 "2023-05-11T17:57:32Z")
Hi @DavidTurner Thanks for the reply!! Can you also please provide some insights on how to identify other process which can change data directory? Just to add, its a 3-node cluster with each node running on separate worker nodes & this issue occurs generally after 10-12 days of running cluster ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 12, 2023, 1:17pm UTC](https://discuss.elastic.co/t/3-node-elasticsearch-cluster-is-failing-repeatedly-with-error-this-node-is-unhealthy-health-check-failed-due-to-broken-node-lock/333234/4 "2023-05-12T13:17:16Z")
Common culprits include misconfigured/buggy security scanners and backup tools, but it could be anything really. You'll need to work with your local sysadmin folks to pin it down. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 9, 2023, 1:17pm UTC](https://discuss.elastic.co/t/3-node-elasticsearch-cluster-is-failing-repeatedly-with-error-this-node-is-unhealthy-health-check-failed-due-to-broken-node-lock/333234/5 "2023-06-09T13:17:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.