# Disk Watermark issues with 1.4.0

**URL:** <https://discuss.elastic.co/t/disk-watermark-issues-with-1-4-0/21009>\
**Category:** Elasticsearch\
**Created:** [December 1, 2014, 4:28pm UTC](https://discuss.elastic.co/t/disk-watermark-issues-with-1-4-0/21009 "2014-12-01T16:28:08Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Chris\_Neal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_neal/32/3527_2.png) [@Chris\_Neal](https://discuss.elastic.co/u/Chris_Neal)\
**Post date:** [December 1, 2014, 4:28pm UTC](https://discuss.elastic.co/t/disk-watermark-issues-with-1-4-0/21009/1 "2014-12-01T16:28:08Z")

</div>

Hi all,

I'm running 1.4.0. and using the default settings for:

cluster.routing.allocation.disk.watermark.low  
and  
cluster.routing.allocation.disk.watermark.high

I hit an OOME which caused me to need to cycle a node, and then all shards  
that should live on that node stayed unallocated once I brought it back up.

There was no notification anywhere that I had hit any dis space limits, at  
least that I could find. I tried cycling again, nothing. It wasn't until  
I tried to manually reroute one of the shards that I got an indication of  
what was going on:

root@ip-10-0-0-45:bddevw07[1038]:~\> ./reroute  
{"error":"RemoteTransportException[[elasticsearch-ip-10-0-0-12][inet[/10.0.0.12:9300]][cluster:admin/reroute]];  
nested: ElasticsearchIllegalArgumentException[[allocate] allocation of  
[derbysoft-20141130][0] on node  
[elasticsearch-ip-10-0-0-45][Li1yyXUHR8qQn6QHCSahCg][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9300]]{master=true}  
is not allowed, reason: [YES(shard is not allocated to same node or  
host)][YES(node passes include/exclude/require filters)][YES(primary is  
already active)][YES(below shard recovery limit of [2])][YES(allocation  
disabling is ignored)][YES(allocation disabling is ignored)][YES(no  
allocation awareness enabled)][YES(total shard limit disabled: [-1] \<=  
0)][YES(target node version [1.4.0] is same or newer than source node  
version [1.4.0])][NO(less than required [15.0%] free disk on node, free:  
[15.0%])][YES(shard not primary or relocation disabled)]]; ","status":400}

Then I cleaned up some disk space, but there was no auto re-allocation  
afterwards. Once I again tried to manually re-route a shard, then ALL of  
them began rerouting.

My questions are:

- Is there a notification log message somewhere that I missed that would  
have let me know what was going on? If not, there sure should be!
- Should the shard allocation process have started automatically once I  
got the disk space issue resolved?

Thanks!  
Chris

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAND3DphhmWn-amiDBrmYi4rB\_tYZa7%3Dn2M9PF5jVY%3DfhPTqMpg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAND3DphhmWn-amiDBrmYi4rB_tYZa7%3Dn2M9PF5jVY%3DfhPTqMpg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [December 1, 2014, 4:35pm UTC](https://discuss.elastic.co/t/disk-watermark-issues-with-1-4-0/21009/2 "2014-12-01T16:35:40Z")

</div>

On Mon, Dec 1, 2014 at 11:28 AM, Chris Neal [chris.neal@derbysoft.net](mailto:chris.neal@derbysoft.net)  
wrote:

> Hi all,
> 
> I'm running 1.4.0. and using the default settings for:
> 
> cluster.routing.allocation.disk.watermark.low  
> and  
> cluster.routing.allocation.disk.watermark.high
> 
> I hit an OOME which caused me to need to cycle a node, and then all shards  
> that should live on that node stayed unallocated once I brought it back up.
> 
> There was no notification anywhere that I had hit any dis space limits, at  
> least that I could find. I tried cycling again, nothing. It wasn't until  
> I tried to manually reroute one of the shards that I got an indication of  
> what was going on:
> 
> root@ip-10-0-0-45:bddevw07[1038]:~\> ./reroute
> 
> {"error":"RemoteTransportException[[elasticsearch-ip-10-0-0-12][inet[/10.0.0.12:9300]][cluster:admin/reroute]];  
> nested: ElasticsearchIllegalArgumentException[[allocate] allocation of  
> [derbysoft-20141130][0] on node  
> [elasticsearch-ip-10-0-0-45][Li1yyXUHR8qQn6QHCSahCg][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9300]]{master=true}  
> is not allowed, reason: [YES(shard is not allocated to same node or  
> host)][YES(node passes include/exclude/require filters)][YES(primary is  
> already active)][YES(below shard recovery limit of [2])][YES(allocation  
> disabling is ignored)][YES(allocation disabling is ignored)][YES(no  
> allocation awareness enabled)][YES(total shard limit disabled: [-1] \<=  
> 0)][YES(target node version [1.4.0] is same or newer than source node  
> version [1.4.0])][NO(less than required [15.0%] free disk on node, free:  
> [15.0%])][YES(shard not primary or relocation disabled)]]; ","status":400}
> 
> Then I cleaned up some disk space, but there was no auto re-allocation  
> afterwards. Once I again tried to manually re-route a shard, then ALL of  
> them began rerouting.
> 
> My questions are:
> 
> - Is there a notification log message somewhere that I missed that  
> would have let me know what was going on? If not, there sure should be!

A WARN query log every 30 seconds was added in the very last release.

> - 
> - Should the shard allocation process have started automatically once  
> I got the disk space issue resolved?

If you have unallocated shards it should kick in after a few seconds. It  
takes a few seconds for the cluster to notice the change in disk free. If  
there aren't unallocated shards I've sometime found that I need to manually  
shift a shard around to prime the pump. I'm not sure if that has been  
fixed recently though.

I don't think that disk space should prevent a shard from coming up on a  
node that already has it though. I imagine that depends on how much data  
has to be copied to that node but I'm not sure.

Nik

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2YSZo4%3D2\_J2quT\_aR3F\_SGH9p8WPEf-uMUy1W52H-L1g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2YSZo4%3D2_J2quT_aR3F_SGH9p8WPEf-uMUy1W52H-L1g%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Chris\_Neal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_neal/32/3527_2.png) [@Chris\_Neal](https://discuss.elastic.co/u/Chris_Neal)\
**Post date:** [December 1, 2014, 4:41pm UTC](https://discuss.elastic.co/t/disk-watermark-issues-with-1-4-0/21009/3 "2014-12-01T16:41:37Z")

</div>

Thanks for the quick reply Nik 🙂

I've got updating to 1.4.1 on my TODO list for today, as I see there were  
some updates in the Release notes pertaining to this as well. I might let  
things fill up again in Dev and see what happens.

Maybe I wasn't patient enough for the rerouting to start on its own. It  
seems like I waited several minutes before I did it manually, but I'll pay  
more attention the next time.

Thanks again for the input.  
Chris

On Mon, Dec 1, 2014 at 10:35 AM, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com) wrote:

> On Mon, Dec 1, 2014 at 11:28 AM, Chris Neal [chris.neal@derbysoft.net](mailto:chris.neal@derbysoft.net)  
> wrote:
> 
> > Hi all,
> > 
> > I'm running 1.4.0. and using the default settings for:
> > 
> > cluster.routing.allocation.disk.watermark.low  
> > and  
> > cluster.routing.allocation.disk.watermark.high
> > 
> > I hit an OOME which caused me to need to cycle a node, and then all  
> > shards that should live on that node stayed unallocated once I brought it  
> > back up.
> > 
> > There was no notification anywhere that I had hit any dis space limits,  
> > at least that I could find. I tried cycling again, nothing. It wasn't  
> > until I tried to manually reroute one of the shards that I got an  
> > indication of what was going on:
> > 
> > root@ip-10-0-0-45:bddevw07[1038]:~\> ./reroute
> > 
> > {"error":"RemoteTransportException[[elasticsearch-ip-10-0-0-12][inet[/10.0.0.12:9300]][cluster:admin/reroute]];  
> > nested: ElasticsearchIllegalArgumentException[[allocate] allocation of  
> > [derbysoft-20141130][0] on node  
> > [elasticsearch-ip-10-0-0-45][Li1yyXUHR8qQn6QHCSahCg][ip-10-0-0-45.us-west-2.compute.internal][inet[ip-10-0-0-45.us-west-2.compute.internal/10.0.0.45:9300]]{master=true}  
> > is not allowed, reason: [YES(shard is not allocated to same node or  
> > host)][YES(node passes include/exclude/require filters)][YES(primary is  
> > already active)][YES(below shard recovery limit of [2])][YES(allocation  
> > disabling is ignored)][YES(allocation disabling is ignored)][YES(no  
> > allocation awareness enabled)][YES(total shard limit disabled: [-1] \<=  
> > 0)][YES(target node version [1.4.0] is same or newer than source node  
> > version [1.4.0])][NO(less than required [15.0%] free disk on node, free:  
> > [15.0%])][YES(shard not primary or relocation disabled)]];  
> > ","status":400}
> > 
> > Then I cleaned up some disk space, but there was no auto re-allocation  
> > afterwards. Once I again tried to manually re-route a shard, then ALL of  
> > them began rerouting.
> > 
> > My questions are:
> > 
> > - Is there a notification log message somewhere that I missed that  
> > would have let me know what was going on? If not, there sure should be!
> 
> A WARN query log every 30 seconds was added in the very last release.
> 
> > - 
> > - Should the shard allocation process have started automatically once  
> > I got the disk space issue resolved?
> 
> If you have unallocated shards it should kick in after a few seconds. It  
> takes a few seconds for the cluster to notice the change in disk free. If  
> there aren't unallocated shards I've sometime found that I need to manually  
> shift a shard around to prime the pump. I'm not sure if that has been  
> fixed recently though.
> 
> I don't think that disk space should prevent a shard from coming up on a  
> node that already has it though. I imagine that depends on how much data  
> has to be copied to that node but I'm not sure.
> 
> Nik
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2YSZo4%3D2\_J2quT\_aR3F\_SGH9p8WPEf-uMUy1W52H-L1g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2YSZo4%3D2_J2quT_aR3F_SGH9p8WPEf-uMUy1W52H-L1g%40mail.gmail.com)  
> [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2YSZo4%3D2\_J2quT\_aR3F\_SGH9p8WPEf-uMUy1W52H-L1g%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2YSZo4%3D2_J2quT_aR3F_SGH9p8WPEf-uMUy1W52H-L1g%40mail.gmail.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAND3DpgLsPhJBXWFUV8SNN8LR%2ByTdKrFgf6LpceyfDddxqCvxg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAND3DpgLsPhJBXWFUV8SNN8LR%2ByTdKrFgf6LpceyfDddxqCvxg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:46am UTC](https://discuss.elastic.co/t/disk-watermark-issues-with-1-4-0/21009/4 "2017-07-06T00:46:46Z")

</div>

Closed on July 6, 2017, 12:46am UTC