# Node sync fails and cluster goes to "red"

**URL:** <https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409>\
**Category:** Elasticsearch\
**Created:** [May 29, 2021, 7:27pm UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409 "2021-05-29T19:27:54Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![parthmaniar](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\
**Post date:** [May 29, 2021, 7:27pm UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409/1 "2021-05-29T19:27:54Z")

</div>

Hello,

I hope you and your loved ones are safe and healthy.

I have a 3 node cluster with 2 data nodes and 1 voting only node.

I accidentally **deleted the separate disk** that holds data for one of the data node. I separate out data folder for elasticsearch and OS/Installation folder on different disks.

**Prior to the accident cluster was in** `green` **state.**

I have reattached a new disk to the VM but the synchronization fails at 96% and the `elasticsearch service on the secondary node that has all of the data` crashes.

**This cluseter holds two years worth of research data as part of my reading for MSc degree and I am hoping not to lose the data.**

**I am not panickig since one of the nodes has all of the data.**

However the cluser is not synchornising and hence inoperational to collect additional data.

Largest shard is of 100 GB and most of the research data is in shards of ~70GB. There are 10 such shards

**I have:**

1. Stopped all ingestion to the cluster.
2. Stopped all other services (kibana, heartbeat, metricbeat) on the nodes
3. All logs are ingested via a seperate node running logstash which is shutdown.
4. Rebooted both of the data nodes.
5. Shutdown the voting node, which lead to this error when trying to query primary node using postman:

```auto
"type": "master_not_discovered_exception",

```

1. Restarted the voting only node.
2. Increased RAM to the VMs at 48 GB per node and 24 GB to Elasticsearch through jvm.opions
3. Set `sysctl -w vm.max_map_count=262144`

**Last log entry on the secondary node (the one that is holding all of the data) is:**

```auto
[2021-05-29T18:47:07,958][INFO][o.e.x.s.s.SecurityStatusChangeListener] [secondarynode] Active license is now [TRIAL]; Security is enabled
[2021-05-29T18:47:07,964][INFO][o.e.h.AbstractHttpServerTransport] [secondarynode] publish_address {192.168.0.236:9200}, bound_addresses {192.168.0.236:9200}
[2021-05-29T18:47:07,964][INFO][o.e.n.Node] [secondarynode] started
[2021-05-29T18:47:51,696][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [secondarynode] fatal error in thread [elasticsearch[secondarynode][generic][T#12]], exiting
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
        at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:679) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.indices.recovery.RemoteRecoveryTargetHandler$1.tryAction(RemoteRecoveryTargetHandler.java:235) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.action.support.RetryableAction$1.doRun(RetryableAction.java:88) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:215) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.action.support.RetryableAction.run(RetryableAction.java:66) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.indices.recovery.RemoteRecoveryTargetHandler.executeRetryableAction(RemoteRecoveryTargetHandler.java:245) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.indices.recovery.RemoteRecoveryTargetHandler.writeFileChunk(RemoteRecoveryTargetHandler.java:205) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.indices.recovery.RecoverySourceHandler$2.executeChunkRequest(RecoverySourceHandler.java:950) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.indices.recovery.RecoverySourceHandler$2.executeChunkRequest(RecoverySourceHandler.java:903) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.indices.recovery.MultiChunkTransfer.handleItems(MultiChunkTransfer.java:112) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.indices.recovery.MultiChunkTransfer.access$000(MultiChunkTransfer.java:48) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.indices.recovery.MultiChunkTransfer$1.write(MultiChunkTransfer.java:67) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.util.concurrent.AsyncIOProcessor.processList(AsyncIOProcessor.java:97) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.util.concurrent.AsyncIOProcessor.drainAndProcessAndRelease(AsyncIOProcessor.java:85) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.util.concurrent.AsyncIOProcessor.put(AsyncIOProcessor.java:76) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.indices.recovery.MultiChunkTransfer.addItem(MultiChunkTransfer.java:78) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.indices.recovery.MultiChunkTransfer.lambda$handleItems$3(MultiChunkTransfer.java:113) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:387) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:387) ~[elasticsearch-7.13.0.jar:7.13.0]
        at 

```

However, the service on the node that holds the data keeps crashing during sync. Here is the log as I see it on the primary node (the one where I deleted the data disk and is receiving the data).

**Here are the last few lines on the primarynode:**

```auto
[2021-05-29T19:19:56,469][INFO][o.e.i.s.IndexShard] [primarynode] [filebeat-7.12.1-2021.05.18][0] primary-replica resync completed with 0 operations
[2021-05-29T19:19:56,470][INFO][o.e.i.s.IndexShard] [primarynode] [.ml-config][0] primary-replica resync completed with 0 operations
[2021-05-29T19:19:56,485][INFO][o.e.c.r.DelayedAllocationService] [primarynode] scheduling reroute for delayed shards in [58.7s] (629 delayed shards)
[2021-05-29T19:19:56,934][WARN][o.e.a.b.TransportShardBulkAction] [primarynode] [[packetbeat-7.13.0-2021.05.28-000001][0]] failed to perform indices:data/write/bulk[s] on replica [packetbeat-7.13.0-2021.05.28-000001][0], node[DCEyrCsJSw6xVPw0xFpO5Q], [R], s[STARTED], a[id=Yup1YJgXQbKWWPl_fdxS9g]
org.elasticsearch.client.transport.NoNodeAvailableException: unknown node [DCEyrCsJSw6xVPw0xFpO5Q]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:1070) [elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.action.support.replication.ReplicationOperation$3.tryAction(ReplicationOperation.java:233) [elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.action.support.RetryableAction$1.doRun(RetryableAction.java:88) [elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) [elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.13.0.jar:7.13.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:831) [?:?]
        Suppressed: org.elasticsearch.transport.NodeNotConnectedException: [secondarynode][192.168.0.236:9300] Node not connected
                at org.elasticsearch.transport.ClusterConnectionManager.getConnection(ClusterConnectionManager.java:178) ~[elasticsearch-7.13.0.jar:7.13.0]
                at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:780) ~[elasticsearch-7.13.0.jar:7.13.0]
                at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:679) ~[elasticsearch-7.13.0.jar:7.13.0]
                at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:1077) [elasticsearch-7.13.0.jar:7.13.0]
                at org.elasticsearch.action.support.replication.ReplicationOperation$3.tryAction(ReplicationOperation.java:233) [elasticsearch-7.13.0.jar:7.13.0]
                at org.elasticsearch.action.support.RetryableAction$1.doRun(RetryableAction.java:88) [elasticsearch-7.13.0.jar:7.13.0]
                at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) [elasticsearch-7.13.0.jar:7.13.0]
                at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.13.0.jar:7.13.0]
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
                at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
                at java.lang.Thread.run(Thread.java:831) [?:?]
        Suppressed: org.elasticsearch.transport.NodeNotConnectedException: [secondarynode][192.168.0.236:9300] Node not connected
                at org.elasticsearch.transport.ClusterConnectionManager.getConnection(ClusterConnectionManager.java:178) ~[elasticsearch-7.13.0.jar:7.13.0]
                at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:780) ~[elasticsearch-7.13.0.jar:7.13.0]
                at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:679) ~[elasticsearch-7.13.0.jar:7.13.0]
                at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:1077) [elasticsearch-7.13.0.jar:7.13.0]
                at org.elasticsearch.action.support.replication.ReplicationOperation$3.tryAction(ReplicationOperation.java:233) [elasticsearch-7.13.0.jar:7.13.0]
                at org.elasticsearch.action.support.RetryableAction$1.doRun(RetryableAction.java:88) [elasticsearch-7.13.0.jar:7.13.0]
                at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) [elasticsearch-7.13.0.jar:7.13.0]
                at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.13.0.jar:7.13.0]
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
                at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) 
  
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
                at java.lang.Thread.run(Thread.java:831) [?:?]
[2021-05-29T19:19:56,935][WARN][o.e.c.r.a.AllocationService] [primarynode] [packetbeat-7.13.0-2021.05.28-000001][0] marking unavailable shards as stale: [Yup1YJgXQbKWWPl_fdxS9g]
[2021-05-29T19:19:59,647][WARN][o.e.c.r.a.AllocationService] [primarynode] [winlogbeat-7.13.0-2021.05.28-000001][0] marking unavailable shards as stale: [DlbwBVX2RgO5Ny1XOOpYvQ]

```

**What should I do to make the cluster operational?**

1. **Should I delete the data on the primarynode and attempt a resync?**
2. There are few indices which hold the research data ( **all with name Cowrie-** while others I can recreate, how do I save my research data?)

_Please view my comments to see the diagnostics I've attempted._

---

<div class="post-metadata">

**Author:** ![parthmaniar](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\
**Post date:** [May 29, 2021, 8:13pm UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409/2 "2021-05-29T20:13:14Z")

</div>

Due to character limitations I am continuing here:

When I run : `_cat/indices/*?v&s=pri.store.size:desc` on the **secondary node** , see that only the indices that I do not need are in red (with total of just 4)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/6/067a5afbba9c64a09aa18c3e6d1f0038bbb12790.png)

**And all of my research indices are in Yellow:**

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/e/2ed707179a84b50b309699de46cb803901eaa46e.png)

However this changes when I look at the same data from the **primary node:** My research indices are in red

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/b/6b08870b9a8abd328de9c86eb3db08d013ce5b59.png)

**There are total of 61 indices in red when I query the primarynode**

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 30, 2021, 6:26am UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409/3 "2021-05-30T06:26:35Z")

</div>

> [@parthmaniar](#):
>
> ```auto
> [2021-05-29T18:47:51,696][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [secondarynode] fatal error in thread [elasticsearch[secondarynode][generic][T#12]], exiting
> java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
> 
> ```

My guess would be a faulty disk on this node. Does `dmesg` report any problems?

> [@parthmaniar](#):
>
> I am not panickig since one of the nodes has all of the data.

Do you have snapshots of this data too?

---

<div class="post-metadata">

**Author:** ![parthmaniar](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\
**Post date:** [May 30, 2021, 6:34am UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409/4 "2021-05-30T06:34:44Z")

</div>

Hello David,

Thank you for replying.

I do not have snapshots for the cluster. It unfortunately started with the intention to take snapshots which failed : [https://discuss.elastic.co/t/backup-snapshot-strategy-for-a-two-data-node-cluster/](https://discuss.elastic.co/t/backup-snapshot-strategy-for-a-two-data-node-cluster/)

Here is the dmseg log from the node where the data is available (secondary node) . It in fact has new a SSD I got 2 weeks back 😟

```auto
[9002.852451] sd 32:0:1:0: [sdb] tag#36 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[9002.852462] sd 32:0:1:0: [sdb] tag#36 Sense Key : Hardware Error [current]
[9002.852475] sd 32:0:1:0: [sdb] tag#36 Add. Sense: Internal target failure
[9002.852478] sd 32:0:1:0: [sdb] tag#36 CDB: Read(10) 28 00 21 fa 09 00 00 01 00 00
[9002.852480] blk_update_request: critical target error, dev sdb, sector 570034432 op 0x0:(READ) flags 0x80700 phys_seg 9 prio class 0
[9003.028072] sd 32:0:1:0: [sdb] tag#37 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[9003.028086] sd 32:0:1:0: [sdb] tag#37 Sense Key : Hardware Error [current]
[9003.028089] sd 32:0:1:0: [sdb] tag#37 Add. Sense: Internal target failure
[9003.028091] sd 32:0:1:0: [sdb] tag#37 CDB: Read(10) 28 00 21 fa 0a 00 00 01 00 00
[9003.028094] blk_update_request: critical target error, dev sdb, sector 570034688 op 0x0:(READ) flags 0x80700 phys_seg 9 prio class 0
[9003.294889] sd 32:0:1:0: [sdb] tag#42 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[9003.294898] sd 32:0:1:0: [sdb] tag#42 Sense Key : Hardware Error [current]
[9003.294901] sd 32:0:1:0: [sdb] tag#42 Add. Sense: Internal target failure
[9003.294903] sd 32:0:1:0: [sdb] tag#42 CDB: Read(10) 28 00 21 fa 09 98 00 00 08 00
[9003.294906] blk_update_request: critical target error, dev sdb, sector 570034584 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[9003.467311] sd 32:0:1:0: [sdb] tag#43 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[9003.467324] sd 32:0:1:0: [sdb] tag#43 Sense Key : Hardware Error [current]
[9003.467326] sd 32:0:1:0: [sdb] tag#43 Add. Sense: Internal target failure
[9003.467329] sd 32:0:1:0: [sdb] tag#43 CDB: Read(10) 28 00 21 fa 09 98 00 00 08 00
[9003.467332] blk_update_request: critical target error, dev sdb, sector 570034584 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0

```

Will I be at least able to recovery the cluster with my research data?

Just to add information here, I store OS&Elasticsearch on a single disk and all of the data on a separate physical disk

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/d/4d850d9655c2d0e1010b0c2b1bc45229f98996af.png)

---

<div class="post-metadata">

**Author:** ![csaltos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/csaltos/32/89464_2.png) [@csaltos](https://discuss.elastic.co/u/csaltos)\
**Post date:** [May 30, 2021, 6:41am UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409/5 "2021-05-30T06:41:50Z")

</div>

What about using `bin/elasticsearch-node repurpose` or `bin/elasticsearch-node detach-cluster` and then reattach it ... it's a long shot but maybe it can help

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 30, 2021, 6:43am UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409/6 "2021-05-30T06:43:58Z")

</div>

Ok `sdb` is definitely broken. If you currently have no snapshots then that might mean there are no good copies of at least one shard.

There is no way to know whether this disk will fail further, so I recommend you take a snapshot of as much data as you can right now. You might need to go index-by-index since snapshotting the broken index will likely fail. Send the snapshots to an independent system, something like S3.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 30, 2021, 6:45am UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409/7 "2021-05-30T06:45:38Z")

</div>

> [@csaltos](#):
>
> What about using `bin/elasticsearch-node repurpose` or `bin/elasticsearch-node detach-cluster` and then reattach it ... it's a long shot but maybe it can help

This advice makes no sense, the node does not need repurposing and the cluster is forming fine so detaching is unnecessary. It's a straightforward read error.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 30, 2021, 6:48am UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409/8 "2021-05-30T06:48:06Z")

</div>

Are both data nodes not also master nodes? Are you really running with a dedicated data node in a three node cluster?

---

<div class="post-metadata">

**Author:** ![parthmaniar](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\
**Post date:** [May 30, 2021, 6:48am UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409/9 "2021-05-30T06:48:54Z")

</div>

Thank you for that David.

I had attached an iSCSI volume to check if I can take the snapshot. I feel this error would be of that time. It was attached to sdb.

Secondarynode which has the data runs fine without the primary node. It shows `all indices in yellow` except `two in red which were created yesterday.` I am OK to lose this indices.

Can I do integrity validation on the indices to see which one is having the corrupt sectors?

Also, can I backup the cluster settings and just the indices I want (Cowrie-\*) to S3 as opposed to all the indices and restore on the primary node?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 30, 2021, 6:53am UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409/10 "2021-05-30T06:53:34Z")

</div>

> [@parthmaniar](#):
>
> I had attached an iSCSI volume to check if I can take the snapshot.

I strongly recommend taking snapshots to an independent system like S3. Taking a snapshot to another potentially broken disk won't help.

> [@parthmaniar](#):
>
> Can I do integrity validation on the indices to see which one is having the corrupt sectors?

Taking a snapshot includes integrity validation. Work index-by-index until you find the one that fails.

> [@parthmaniar](#):
>
> Also, can I backup the cluster settings and just the indices I want (Cowrie-\*) to S3 as opposed to all the indices and restore on the primary node?

Yes, you can (and will need to) snapshot a single index at a time.

---

<div class="post-metadata">

**Author:** ![parthmaniar](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\
**Post date:** [May 30, 2021, 6:53am UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409/11 "2021-05-30T06:53:55Z")

</div>

Hello Christian:

Here is the setting from elasticsearch.yml for the `secondary node (192.168.0.236)` that is having the data intact:

```auto
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#discovery.seed_hosts: ["primarynode", "secondarynode"]
#discovery.seed_hosts: ["primarynode", "secondarynode", "votingonlynode"]
discovery.seed_hosts: ["192.168.0.235", "192.168.0.236", "192.168.0.237"]
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#cluster.initial_master_nodes: ["primarynode", "secondarynode"]

#30-05-2021: Commented out on the primarynode as part of diagnosing sync failure because I deleted data disk on the primarynode. Entry noted here for consistency. 
cluster.initial_master_nodes: ["192.168.0.235", "192.168.0.236"]

# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
node.master: true
node.voting_only: false

```

I had a community member ask me to carry out the following to try and restore but it fails at writing 300 GB to the `primarynode` that I am trying to sync data to:

1. Rename \_state to \_state\_backup (anything but \_state)
2. Comment out: `cluster.initial_master_nodes: ["192.168.0.235", "192.168.0.236"]` and reboot the cluster.

Yes there are 2 data nodes and one voting node in the cluster.

**This did not solve the problem**

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 30, 2021, 6:56am UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409/12 "2021-05-30T06:56:05Z")

</div>

> [@parthmaniar](#):
>
> I had a community member ask me to carry out the following

They gave you some very bad advice. You should never manipulate the contents of the data path yourself.

(The advice to remove `cluster.initial_master_nodes` is sound, you should do that anyway, but I don't think it's relevant here)

---

<div class="post-metadata">

**Author:** ![parthmaniar](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\
**Post date:** [May 30, 2021, 7:06am UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409/13 "2021-05-30T07:06:47Z")

</div>

Thanks David. Thank you for the advise, in my case data.path and data.logs both were on separate disk that on a VM that I deleted by mistake so I reckon they were empty and waiting for be synced.

I suspect disk errors would have come when I attached a new disk which took up /dev/sdb.

I've run badblocks to see if I can get additional data.

should I remove `cluster.initial_master_nodes` from all the nodes now or wait for full recovery before thouching those settings?

Could I ask you to have a look at this: [Recent unsafe memory access operation in compiled Java code - #7 by jprante](https://discuss.elastic.co/t/recent-unsafe-memory-access-operation-in-compiled-java-code/41372/7). I feel it maybe the size of the indices that is causing the sync issue?

---

<div class="post-metadata">

**Author:** ![csaltos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/csaltos/32/89464_2.png) [@csaltos](https://discuss.elastic.co/u/csaltos)\
**Post date:** [May 30, 2021, 7:09am UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409/14 "2021-05-30T07:09:22Z")

</div>

I knew it was a long shot ... thank for taking your time for correcting me.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 30, 2021, 7:15am UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409/15 "2021-05-30T07:15:46Z")

</div>

> [@parthmaniar](#):
>
> should I remove `cluster.initial_master_nodes` from all the nodes now

Up to you, there's no rush, it's nothing to do with your current problems.

> [@parthmaniar](#):
>
> I feel it maybe the size of the indices that is causing the sync issue?

No, it's a faulty disk, it could affect an index of any size. I suppose a bad block is more likely to affect a larger index just because larger indices have more blocks.

The post you linked seems quite confused, we don't use mmap to write anything, we never require having the whole file in memory at once anyway, and even if we did it wouldn't result in errors like this when running out of space.

---

<div class="post-metadata">

**Author:** ![parthmaniar](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\
**Post date:** [May 30, 2021, 7:26am UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409/16 "2021-05-30T07:26:30Z")

</div>

Thank you very much for the extended support David. I deeply appreciate it.

While I did not doubt you, the fault SSD seems to be correct! Because I recently added this disk and attached it to the VM. At that time the data synced without any issues! I should have thought of it (just a testament that I need to take a break from studies and work).

I've run `badlocks` on the disk and I will pass it to `fsck` to help while the replacement disk comes in.

I'll try and delete larger indices which I don't need with the hope that one of them had the bad block(s). I'll try and sync the nodes after this. Hopefully it will work. My research work is almost 750 GB in size split into 15 indices. I will try and snapshot them to S3 but I am not sure given my home internet connection upload speed (capped at 5 Mbps).

Keeping my fingers crossed.

_With all the resiliency planned (separating OS/Application and data into separate physical disks) one error in fatigue (of deleting the virtual disk on the primary node) has caused sincere pain to me. Exhaustion / burnout is real. This has been a learning._

---

<div class="post-metadata">

**Author:** ![parthmaniar](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\
**Post date:** [May 30, 2021, 8:20am UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409/17 "2021-05-30T08:20:31Z")

</div>

Hello @DavidTurner , is it safe to delete following indices which are large in size and I reckon they do not have the raw data but analysed data which I can recreate?

```auto
green open .monitoring-es-7-2021.05.25 Z87RDUcyQZqiFOiL-huaIw 1 0 10089695 1249720 3.9gb 3.9gb
green open .monitoring-es-7-2021.05.24 8yjVzoSJQkSL1YYce_tdkA 1 0 9922505 1331520 3.8gb 3.8gb
green open .monitoring-es-7-mb-2021.05.28 WecYTFQoSgeqY3L1H1gWig 1 0 6966750 0 3.8gb 3.8gb
green open .monitoring-es-7-2021.05.27 SCWv7GjkRKi51ozluCLMpA 1 0 10503113 991852 3.7gb 3.7gb
green open .monitoring-es-7-2021.05.26 CW_pa3xLTieP8sTtQhFihA 1 0 9978777 1228898 3.6gb 3.6gb
green open .monitoring-es-7-mb-2021.05.29 -gzR63-YSTm5nAAYMnawGA 1 0 7348808 0 3.4gb 3.4gb
green open .monitoring-es-7-mb-2021.05.26 kMGnzT9hTjCuzQFKQBgwtQ 1 0 5195177 0 2.8gb 2.8gb

```

I've deleted most of the large indices except the that store the raw research data. Hopefully this will work out now 🙂

If it does I will apply leave at for entire week 😃 & just relax.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [May 30, 2021, 8:48am UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409/18 "2021-05-30T08:48:33Z")

</div>

Their names suggest that these indices contain monitoring data about your cluster. I doubt you can re-create it but it's up to you whether you need to keep it or not.

However I strongly recommend you focus on making a copy of your important data elsewhere first. Disk failures can be progressive so the longer you delay the bigger the risk you will lose more data. It's possible that deleting data will cause other data to move around, potentially causing more of it to end up written to unreadable blocks. Moreover I can't think of a mechanism by which deleting some unimportant data will help preserve any of the important data. This path seems only to have downsides.

---

<div class="post-metadata">

**Author:** ![parthmaniar](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\
**Post date:** [May 30, 2021, 12:40pm UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409/19 "2021-05-30T12:40:45Z")

</div>

Hello,

I'm very `happy` to report that the indices holding my research data are not on bad sectors.

I have successfully been able to sync data to primary node (and hence other SSD).

I am now going to take snapshot on AWS (or Linode since cost calculator for AWS is at 50 USD 😟 ). OR on my NAS using NFS mount (NAS has hardware layer redundancy)

**Second** : I can see cluster still being RED because single indice does not have primary or replica shards. I am sure it is lost, I am unable to delete it. The size is 0 if I query elasticsearch. I cannot see inside Kibana GUI. -- **Is there a way to resolve this?**

```auto
red open .ds-ilm-history-5-2021.05.13-000004 o2ew_zK5RGSgP-binaUbyw 1 1                                                

```

I have rebuild my dashboards and other components, which I reckon is my only choice - **backup and test your backups is a learning**

Once the snapshots are done, I will move the storage of Elasticsearch data to another SSD.

Thank you to @DavidTurner@Christian_Dahlqvist and @csaltos for the extended help. Have a wonderful time ahead.

---

<div class="post-metadata">

**Author:** ![parthmaniar](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\
**Post date:** [May 30, 2021, 5:48pm UTC](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409/20 "2021-05-30T17:48:40Z")

</div>

Hello,

I've mounted a NFS share & **added it as a repository** on the data nodes in the cluster and checked its connectivity:

![image](https://us1.discourse-cdn.com/elastic/original/3X/a/f/af707ef8174a5fb12d97e036b5aff0c0e44d0f02.png)

Backups are however **failing** unless I keep following on `Ignore unavailable indices` and `Allow partial indices` turned on. Error:

```auto
{
  "type": "snapshot_exception",
  "reason": "[nas_backup:daily-full--dpmpzql9t2g0qla24hpxtq/eNAtkTMQSVmAeyaPcVIe1A] Indices don't have primary shards [.ds-ilm-history-5-2021.05.13-000004]",
  "stack_trace": "SnapshotException[[nas_backup:daily-full--dpmpzql9t2g0qla24hpxtq/eNAtkTMQSVmAeyaPcVIe1A] Indices don't have primary shards [.ds-ilm-history-5-2021.05.13-000004]]\n\tat org.elasticsearch.snapshots.SnapshotsService$2.execute(SnapshotsService.java:491)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository$1.execute(BlobStoreRepository.java:393)\n\tat org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:48)\n\tat org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:691)\n\tat org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:313)\n\tat org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:208)\n\tat org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:62)\n\tat org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:140)\n\tat org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:139)\n\tat org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:177)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:241)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:204)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:831)\n"
}

```

**I've set individual policy just to backup the research data. It is ongoing. I will update on its completion status soon.**

[Next page](https://discuss.elastic.co/t/node-sync-fails-and-cluster-goes-to-red/274409.md?page=2)