# Cluster is down on 2 nodes server high disk watermark

**URL:** <https://discuss.elastic.co/t/cluster-is-down-on-2-nodes-server-high-disk-watermark/246859>\
**Category:** Elasticsearch\
**Created:** [August 29, 2020, 9:30pm UTC](https://discuss.elastic.co/t/cluster-is-down-on-2-nodes-server-high-disk-watermark/246859 "2020-08-29T21:30:32Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dilip\_Kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dilip_kumar/32/15459_2.png) [@Dilip\_Kumar](https://discuss.elastic.co/u/Dilip_Kumar)\
**Post date:** [August 29, 2020, 9:30pm UTC](https://discuss.elastic.co/t/cluster-is-down-on-2-nodes-server-high-disk-watermark/246859/1 "2020-08-29T21:30:32Z")

</div>

Hi All,  
I have two node cluster suddenly cluster got down. es version 5.6 and ram size of each node is 16 GB and heap size is given 8GB. I am sharing my nod1 log , please help to find solution..

```auto
[2020-08-30T00:30:09,917][WARN][o.e.c.r.a.DiskThresholdMonitor] [my_prodnode1] high disk watermark [90%] exceeded on [JCHa7NT_TBuEGi-5Sy7cpQ][my_prodnode2][/var/lib/elasticsearch/nodes/0] free: 204kb[0%], shards will be relocated away from this node
		

[2020-08-29T10:00:05,125][INFO][o.e.m.j.JvmGcMonitorService] [my_prodnode1] [gc][3106046] overhead, spent [268ms] collecting in the last [1s]
[2020-08-29T10:00:06,126][INFO][o.e.m.j.JvmGcMonitorService] [my_prodnode1] [gc][3106047] overhead, spent [274ms] collecting in the last [1s]

[2020-08-29T20:29:01,104][DEBUG][o.e.a.b.TransportShardBulkAction] [my_prodnode1] [myregindex1][0] failed to execute bulk item (update) BulkShardRequest [[myregindex1][0]] containing [org.elasticsearch.action.update.UpdateRequest@5f12f396]
org.elasticsearch.index.engine.DocumentMissingException: [znl][E692DFD2-6CB1-4DF6-91E0-82E50325B31B]: document missing

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_241]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_241]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]

[2020-08-29T23:50:05,006][WARN][o.e.c.a.s.ShardStateAction] [my_prodnode1] [esendzlist][1] received shard failed for shard id [[esendzlist][1]], allocation id [DJWCJzJZQ6esChwDe6zZFA], primary term [0], message [shard failure, reason [refresh failed]], failure [IOException[No space left on device]]

[2020-08-29T23:50:12,282][DEBUG][o.e.a.b.TransportShardBulkAction] [my_prodnode1] [esendzlist][1] failed to execute bulk item (index) BulkShardRequest [[esendzlist][1]] containing [index {[esendzlist][ezlist][80_ESEND], source[n/a, actual length: [2.3mb], max length: 2kb]}]
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [zlist]
        at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:298) ~[elasticsearch-5.6.16.jar:5.6.16]

[2020-08-30T01:03:18,216][WARN][o.e.c.r.a.DiskThresholdMonitor] [my_prodnode1] high disk watermark [90%] exceeded on [JCHa7NT_TBuEGi-5Sy7cpQ][my_prodnode2][/var/lib/elasticsearch/nodes/0] free: 192kb[0%], shards will be relocated away from this node
[2020-08-30T01:03:18,216][WARN][o.e.c.r.a.DiskThresholdMonitor] [my_prodnode1] high disk watermark [90%] exceeded on 

[2020-08-30T01:04:01,405][WARN][o.e.i.e.Engine] [my_prodnode1] [sdkversion][1] failed engine [merge failed]
org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: No space left on device

[2020-08-30T02:11:40,451][INFO][o.e.m.j.JvmGcMonitorService] [my_prodnode1] [gc][22] overhead, spent [325ms] collecting in the last [1s]
[2020-08-30T02:11:50,232][DEBUG][o.e.a.s.TransportSearchAction] [my_prodnode1] All shards failed for phase: [query]
[2020-08-30T02:11:50,233][WARN][r.suppressed] path: /myregindex1/znl/_search, params: {index=myregindex1, type=znl}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed

```

---

<div class="post-metadata">

**Author:** ![Steve\_Mushero](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_mushero/32/22441_2.png) [@Steve\_Mushero](https://discuss.elastic.co/u/Steve_Mushero)\
**Post date:** [August 30, 2020, 2:27am UTC](https://discuss.elastic.co/t/cluster-is-down-on-2-nodes-server-high-disk-watermark/246859/2 "2020-08-30T02:27:51Z")

</div>

> [@Dilip\_Kumar](#):
>
> `java.io.IOException: No space left on device`

Sure looks like you are out of disk space ... only 192KB.

---

<div class="post-metadata">

**Author:** ![Dilip\_Kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dilip_kumar/32/15459_2.png) [@Dilip\_Kumar](https://discuss.elastic.co/u/Dilip_Kumar)\
**Post date:** [August 31, 2020, 3:40am UTC](https://discuss.elastic.co/t/cluster-is-down-on-2-nodes-server-high-disk-watermark/246859/3 "2020-08-31T03:40:41Z")

</div>

Thanks Steve, but hard disk having enough space , are you talking about ram space, currently ram size is 16 GB and heap is given 8GB, so i need to increase ram?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 31, 2020, 4:11am UTC](https://discuss.elastic.co/t/cluster-is-down-on-2-nodes-server-high-disk-watermark/246859/4 "2020-08-31T04:11:52Z")

</div>

> [@Dilip\_Kumar](#):
>
> ```auto
> org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: No space left on device
> 
> ```

Elasticsearch is definitely seeing that there's not much space left.

What does `df -h` show?

---

<div class="post-metadata">

**Author:** ![Dilip\_Kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dilip_kumar/32/15459_2.png) [@Dilip\_Kumar](https://discuss.elastic.co/u/Dilip_Kumar)\
**Post date:** [August 31, 2020, 4:29am UTC](https://discuss.elastic.co/t/cluster-is-down-on-2-nodes-server-high-disk-watermark/246859/5 "2020-08-31T04:29:04Z")

</div>

hi warkolm please find below both node df -h status

```auto
**node 1:**
Filesystem Size Used Avail Use% Mounted on
udev 7.9G 0 7.9G 0% /dev
tmpfs 1.6G 169M 1.4G 11% /run
/dev/sda4 52G 2.0G 48G 4% /
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/sda3 465M 50M 387M 12% /boot
/dev/sda1 19G 783M 17G 5% /var
/dev/mapper/vg_u01-lv_u01 492G 8.7G 458G 2% /u01
/dev/sdc 2.0T 18G 1.9T 1% /nfs
tmpfs 1.6G 0 1.6G 0% /run/user/1002
tmpfs 1.6G 0 1.6G 0% /run/user/1003

**Node2**
Filesystem Size Used Avail Use% Mounted on
udev 7.9G 0 7.9G 0% /dev
tmpfs 1.6G 169M 1.4G 11% /run
/dev/sda4 52G 2.0G 48G 4% /
tmpfs 7.9G 0 7.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/sda1 19G 724M 17G 5% /var
/dev/sda3 465M 50M 387M 12% /boot
/dev/mapper/vg_u01-lv_u01 492G 8.8G 458G 2% /u01
10.201.201.63:/nfs 2.0T 18G 1.9T 1% /nfs
tmpfs 1.6G 0 1.6G 0% /run/user/1002
tmpfs 1.6G 0 1.6G 0% /run/user/1003

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 31, 2020, 4:32am UTC](https://discuss.elastic.co/t/cluster-is-down-on-2-nodes-server-high-disk-watermark/246859/6 "2020-08-31T04:32:23Z")

</div>

Did you change the default `path.data`?

---

<div class="post-metadata">

**Author:** ![Dilip\_Kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dilip_kumar/32/15459_2.png) [@Dilip\_Kumar](https://discuss.elastic.co/u/Dilip_Kumar)\
**Post date:** [August 31, 2020, 4:32am UTC](https://discuss.elastic.co/t/cluster-is-down-on-2-nodes-server-high-disk-watermark/246859/7 "2020-08-31T04:32:44Z")

</div>

also getting

```auto
[2020-08-31T05:02:26,732][INFO][o.e.m.j.JvmGcMonitorService] [my_prodnode2] [gc][97025] overhead, spent [323ms] collecting in the last [1s]
[2020-08-31T05:02:27,743][INFO][o.e.m.j.JvmGcMonitorService] [my_prodnode2] [gc][97026] overhead, spent [260ms] collecting in the last [1s]
[2020-08-31T05:02:29,756][INFO][o.e.m.j.JvmGcMonitorService] [my_prodnode2] [gc][97028] overhead, spent [322ms] collecting in the last [1s]
[2020-08-31T05:02:30,756][INFO][o.e.m.j.JvmGcMonitorService] [my_prodnode2] [gc][97029] overhead, spent [331ms] collecting in the last [1s]
[2020-08-31T05:02:31,759][INFO][o.e.m.j.JvmGcMonitorService] [my_prodnode2] [gc][97030] overhead, spent [329ms] collecting in the last [1s]
[2020-08-31T05:02:32,769][INFO][o.e.m.j.JvmGcMonitorService] [my_prodnode2] [gc][97031] overhead, spent [320ms] collecting in the last [1s]

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 31, 2020, 4:33am UTC](https://discuss.elastic.co/t/cluster-is-down-on-2-nodes-server-high-disk-watermark/246859/8 "2020-08-31T04:33:47Z")

</div>

> [@Dilip\_Kumar](#):
>
> also getting

That is fine, as it says it's an info, not a warn.

---

<div class="post-metadata">

**Author:** ![Dilip\_Kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dilip_kumar/32/15459_2.png) [@Dilip\_Kumar](https://discuss.elastic.co/u/Dilip_Kumar)\
**Post date:** [August 31, 2020, 4:54am UTC](https://discuss.elastic.co/t/cluster-is-down-on-2-nodes-server-high-disk-watermark/246859/9 "2020-08-31T04:54:04Z")

</div>

no , its default setting , for snapshot taken using nfs. memory server giving about RAm of hard disk?

---

<div class="post-metadata">

**Author:** ![dominique.bejean](https://avatars.discourse-cdn.com/v4/letter/d/dc4da7/32.png) [@dominique.bejean](https://discuss.elastic.co/u/dominique.bejean)\
**Post date:** [August 31, 2020, 7:56am UTC](https://discuss.elastic.co/t/cluster-is-down-on-2-nodes-server-high-disk-watermark/246859/10 "2020-08-31T07:56:02Z")

</div>

Hi,

You are running out of disk space. Fix this issue first.  
No space left on device

Regards

Dominique

---

<div class="post-metadata">

**Author:** ![Steve\_Mushero](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_mushero/32/22441_2.png) [@Steve\_Mushero](https://discuss.elastic.co/u/Steve_Mushero)\
**Post date:** [August 31, 2020, 8:13am UTC](https://discuss.elastic.co/t/cluster-is-down-on-2-nodes-server-high-disk-watermark/246859/11 "2020-08-31T08:13:39Z")

</div>

Yeah, but his df -h shows space which is VERY weird.

Could be some temp issue but your /tmp space has space, in fact you have space all over.

Suggest sudo to the ES user and see what it can see, maybe some quota or other weird permission issue, or maybe you are on Docker or have unusual disk setup, even NFS but you are mounted from /dev/sda3 so very odd; I wonder if that's a SAN device or something.

Suggesting making SURE you know your data path and that it has space from that user's perspective.

---

<div class="post-metadata">

**Author:** ![Dilip\_Kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dilip_kumar/32/15459_2.png) [@Dilip\_Kumar](https://discuss.elastic.co/u/Dilip_Kumar)\
**Post date:** [August 31, 2020, 9:53am UTC](https://discuss.elastic.co/t/cluster-is-down-on-2-nodes-server-high-disk-watermark/246859/12 "2020-08-31T09:53:22Z")

</div>

Thanks Steve\_Mushero for your kind attention

```auto
 i am not using any docker, space is already there, i think this pace information coming for RAM .. am i right?

```

---

<div class="post-metadata">

**Author:** ![Steve\_Mushero](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_mushero/32/22441_2.png) [@Steve\_Mushero](https://discuss.elastic.co/u/Steve_Mushero)\
**Post date:** [September 1, 2020, 3:42am UTC](https://discuss.elastic.co/t/cluster-is-down-on-2-nodes-server-high-disk-watermark/246859/13 "2020-09-01T03:42:20Z")

</div>

Suggest trying some of the suggestions like sudo and carefully finding the paths its using, etc. All weird.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 29, 2020, 3:42am UTC](https://discuss.elastic.co/t/cluster-is-down-on-2-nodes-server-high-disk-watermark/246859/14 "2020-09-29T03:42:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.