# High disk watermark exceeded on one or more nodes, rerouting shards
**URL:** https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506
**Category:** Elasticsearch
**Created:** [January 6, 2016, 12:36pm UTC](https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506 "2016-01-06T12:36:01Z")
**Posts on this page:** 18
**Page:** 1
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [January 6, 2016, 12:36pm UTC](https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506/1 "2016-01-06T12:36:01Z")
I am running Elasticsearch, and Kibana on Windows and using Synology NAS as storage for Elasticsearch. For few days, Elasticsearch started behaving weird; therefore, I checked elasticsearch.log and found the following errors:
[WARN][cluster.routing.allocation.decider] [Desmond Pitt] high disk watermark [0b] exceeded on [O2-Ef7fET9S\_MJNAL-q\_yA][Desmond Pitt] free: -1b[100%], shards will be relocated away from this node
[WARN][cluster.routing.allocation.decider] [Desmond Pitt] high disk watermark [0b] exceeded on [O2-Ef7fET9S\_MJNAL-q\_yA][Desmond Pitt] free: -1b[100%], shards will be relocated away from this node
[INFO][cluster.routing.allocation.decider] [Desmond Pitt] high disk watermark exceeded on one or more nodes, rerouting shards
DEBUG][action.bulk] [Desmond Pitt] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
DEBUG][action.bulk] [Desmond Pitt] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
What could be the issue?
---
### Author:  [@bleskes](https://discuss.elastic.co/u/bleskes)
#### Post date: [January 6, 2016, 8:08pm UTC](https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506/2 "2016-01-06T20:08:46Z")
The numbers of 0 bytes and -1b look very suspicious. Maybe the NAS file system is reporting the wrong disk space? When this happens, can check what the `GET _node/stats` returns in terms of disk space?
---
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [January 10, 2016, 4:58am UTC](https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506/3 "2016-01-10T04:58:09Z")
Hi Bleskes,
I am running Elasticsearch on Windows box. Where can I run this command?
---
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [January 10, 2016, 5:04am UTC](https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506/4 "2016-01-10T05:04:30Z")
All right.. I managed to ran to the command. Here is the result:
{"cluster\_name":"elasticsearch","nodes":{"GSqQreOURqmt7tlHf-S2GA":{"timestamp":1
452402067987,"name":"Elektra","transport\_address":"inet[/192.195.88.229:9300]","
host":"tamuq-syslog","ip":["inet[/192.195.88.229:9300]","NONE"],"indices":{"docs
":{"count":191625251,"deleted":0},"store":{"size\_in\_bytes":191255919588,"throttl
e\_time\_in\_millis":1055954},"indexing":{"index\_total":5854665,"index\_time\_in\_mill
is":5035658,"index\_current":4,"delete\_total":0,"delete\_time\_in\_millis":0,"delete
_current":0,"noop\_update\_total":0,"is\_throttled":false,"throttle\_time\_in\_millis"
:0},"get":{"total":2,"time\_in\_millis":38,"exists\_total":2,"exists\_time\_in\_millis
":38,"missing\_total":0,"missing\_time\_in\_millis":0,"current":0},"search":{"open\_c
ontexts":0,"query\_total":435,"query\_time\_in\_millis":191327,"query\_current":0,"fe
tch\_total":6,"fetch\_time\_in\_millis":8589,"fetch\_current":0},"merges":{"current":
1,"current\_docs":4505,"current\_size\_in\_bytes":5058938,"total":17410,"total\_time_
in\_millis":16608728,"total\_docs":51877270,"total\_size\_in\_bytes":58559988274},"re
fresh":{"total":159830,"total\_time\_in\_millis":25448949},"flush":{"total":1548,"t
otal\_time\_in\_millis":353974},"warmer":{"current":0,"total":334721,"total\_time\_in
_millis":1725614},"filter\_cache":{"memory\_size\_in\_bytes":36040,"evictions":0},"i
d\_cache":{"memory\_size\_in\_bytes":0},"fielddata":{"memory\_size\_in\_bytes":44919781
2,"evictions":0},"percolate":{"total":0,"time\_in\_millis":0,"current":0,"memory\_s
ize\_in\_bytes":-1,"memory\_size":"-1b","queries":0},"completion":{"size\_in\_bytes":
0},"segments":{"count":6836,"memory\_in\_bytes":950875584,"index\_writer\_memory\_in_
bytes":580836,"index\_writer\_max\_memory\_in\_bytes":829966741,"version\_map\_memory\_i
n\_bytes":12800,"fixed\_bit\_set\_memory\_in\_bytes":0},"translog":{"operations":332,"
size\_in\_bytes":17},"suggest":{"total":0,"time\_in\_millis":0,"current":0},"query\_c
ache":{"memory\_size\_in\_bytes":0,"evictions":0,"hit\_count":0,"miss\_count":0},"rec
overy":{"current\_as\_source":0,"current\_as\_target":0,"throttle\_time\_in\_millis":0}
},"os":{"timestamp":1452402078885,"uptime\_in\_millis":341884,"cpu":{"sys":19,"use
r":28,"idle":51,"usage":47,"stolen":0},"mem":{"free\_in\_bytes":10495774720,"used\_
in\_bytes":6683623424,"free\_percent":62,"used\_percent":37,"actual\_free\_in\_bytes":
10709585920,"actual\_used\_in\_bytes":6469812224},"swap":{"used\_in\_bytes":615192985
6,"free\_in\_bytes":13577605120}},"process":{"timestamp":1452402078885,"open\_file\_
descriptors":15814,"cpu":{"percent":0,"sys\_in\_millis":141210,"user\_in\_millis":29
5573,"total\_in\_millis":436783},"mem":{"resident\_in\_bytes":270995456,"share\_in\_by
tes":-1,"total\_virtual\_in\_bytes":2431987712}},"jvm":{"timestamp":1452402078885,"
uptime\_in\_millis":262283078,"mem":{"heap\_used\_in\_bytes":3067520784,"heap\_used\_pe
rcent":95,"heap\_committed\_in\_bytes":3203792896,"heap\_max\_in\_bytes":3203792896,"n
on\_heap\_used\_in\_bytes":91917160,"non\_heap\_committed\_in\_bytes":93749248,"pools":{
"young":{"used\_in\_bytes":54359600,"max\_in\_bytes":139591680,"peak\_used\_in\_bytes":
139591680,"peak\_max\_in\_bytes":139591680},"survivor":{"used\_in\_bytes":8907216,"ma
x\_in\_bytes":17432576,"peak\_used\_in\_bytes":17432576,"peak\_max\_in\_bytes":17432576}
,"old":{"used\_in\_bytes":3004253968,"max\_in\_bytes":3046768640,"peak\_used\_in\_bytes
":3013743144,"peak\_max\_in\_bytes":3046768640}}},"threads":{"count":49,"peak\_count
":52},"gc":{"collectors":{"young":{"collection\_count":19821,"collection\_time\_in\_
millis":400711},"old":{"collection\_count":26661,"collection\_time\_in\_millis":1280
905}}},"buffer\_pools":{"direct":{"count":61,"used\_in\_bytes":6511873,"total\_capac
ity\_in\_bytes":6511873},"mapped":{"count":14055,"used\_in\_bytes":190789203170,"tot
al\_capacity\_in\_bytes":190789203170}}},"thread\_pool":{"percolate":{"threads":0,"q
ueue":0,"active":0,"rejected":0,"largest":0,"completed":0},"fetch\_shard\_started"
:{"threads":1,"queue":0,"active":0,"rejected":0,"largest":4,"completed":685},"li
stener":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":1,"completed":3
570},"index":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":0,"complet
ed":0},"refresh":{"threads":1,"queue":0,"active":1,"rejected":0,"largest":1,"com
pleted":159332},"suggest":{"threads":0,"queue":0,"active":0,"rejected":0,"larges
t":0,"completed":0},"generic":{"threads":1,"queue":0,"active":0,"rejected":0,"la
rgest":6,"completed":32671},"warmer":{"threads":1,"queue":0,"active":0,"rejected
":0,"largest":1,"completed":174966},"search":{"threads":4,"queue":0,"active":0,"
rejected":0,"largest":4,"completed":443},"flush":{"threads":1,"queue":0,"active"
:0,"rejected":0,"largest":1,"completed":161214},"optimize":{"threads":0,"queue":
0,"active":0,"rejected":0,"largest":0,"completed":0},"fetch\_shard\_store"
---
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [January 10, 2016, 5:04am UTC](https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506/5 "2016-01-10T05:04:44Z")
Cont.....
:{"threa
ds":0,"queue":0,"active":0,"rejected":0,"largest":0,"completed":0},"management":
{"threads":5,"queue":4,"active":5,"rejected":0,"largest":5,"completed":4141058},
"get":{"threads":2,"queue":0,"active":0,"rejected":0,"largest":2,"completed":2},
"merge":{"threads":1,"queue":0,"active":0,"rejected":0,"largest":1,"completed":1
7427},"bulk":{"threads":2,"queue":0,"active":0,"rejected":0,"largest":2,"complet
ed":432749},"snapshot":{"threads":0,"queue":0,"active":0,"rejected":0,"largest":
0,"completed":0}},"network":{"tcp":{"active\_opens":7278,"passive\_opens":7572,"cu
rr\_estab":69,"in\_segs":291127065,"out\_segs":282045676,"retrans\_segs":1269920,"es
tab\_resets":2582,"attempt\_fails":123,"in\_errs":1,"out\_rsts":3383}},"fs":{"timest
amp":1452402078885,"total":{},"data":[{"path":"\\tamuq-synology1.qatar.tamu.ed
u\syslog\elasticsearch\nodes\0"}]},"transport":{"server\_open":13,"rx\_count":
6,"rx\_size\_in\_bytes":1464,"tx\_count":6,"tx\_size\_in\_bytes":1464},"http":{"current
\_open":2,"total\_opened":11},"breakers":{"fielddata":{"limit\_size\_in\_bytes":19222
75737,"limit\_size":"1.7gb","estimated\_size\_in\_bytes":449197812,"estimated\_size":
"428.3mb","overhead":1.03,"tripped":0},"request":{"limit\_size\_in\_bytes":12815171
58,"limit\_size":"1.1gb","estimated\_size\_in\_bytes":16440,"estimated\_size":"16kb",
"overhead":1.0,"tripped":0},"parent":{"limit\_size\_in\_bytes":2242655027,"limit\_si
ze":"2gb","estimated\_size\_in\_bytes":449214252,"estimated\_size":"428.4mb","overhe
ad":1.0,"tripped":0}}}}}
---
### Author:  [@bleskes](https://discuss.elastic.co/u/bleskes)
#### Post date: [January 11, 2016, 9:26am UTC](https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506/6 "2016-01-11T09:26:31Z")
> [@thyfere](#):
>
> fs":{"timestamp":1452402078885,"total":{},"data":[{"path":"\tamuq-synology1.qatar.tamu.edu\syslog\elasticsearch\nodes\0"}]}
This is indeed what I suspected - the file system fails to report disk usage, which confuses the high water mark check. Can you open an issue about this on github? we should not reroute but rather just log a warning IMO.
---
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [January 11, 2016, 11:04am UTC](https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506/7 "2016-01-11T11:04:05Z")
Thanks Bleskes,
Just to clarify, NAS failed to report disk usage? I have already opened an issue on GitHub but what could be the issue as per your experience?
---
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [January 11, 2016, 2:58pm UTC](https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506/8 "2016-01-11T14:58:01Z")
So, I opened an issue there and here is their reply:
>
They say, it's an Elasticsearch issue.
---
### Author:  [@spalger](https://discuss.elastic.co/u/spalger)
#### Post date: [January 11, 2016, 3:52pm UTC](https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506/9 "2016-01-11T15:52:20Z")
I think @bleskes meant to ask that you file an issue on the [elasticsearch issue tracker](https://github.com/elastic/elasticsearch/issues)
---
### Author:  [@bleskes](https://discuss.elastic.co/u/bleskes)
#### Post date: [January 11, 2016, 6:21pm UTC](https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506/10 "2016-01-11T18:21:12Z")
Sorry. I don’t know what the NAS fails…
---
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [January 13, 2016, 6:41am UTC](https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506/11 "2016-01-13T06:41:13Z")
Hi Bleskes,
What's your experience in regards to attach NAS to Elasticsearch? Does it work normally or are their any hiccups with compare to SAN LUN?
---
### Author:  [@bleskes](https://discuss.elastic.co/u/bleskes)
#### Post date: [January 14, 2016, 5:14pm UTC](https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506/12 "2016-01-14T17:14:22Z")
I can not help with comparing one NAS to another. I can say that using a NAS with ES at all typically leads to poor performance and problems. Remember that ES already has two copies of your data. NAS based redundancy is typically not needed.
---
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [January 20, 2016, 1:00pm UTC](https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506/13 "2016-01-20T13:00:02Z")
Hi Again,
As per [https://github.com/elastic/elasticsearch/issues/16082](https://github.com/elastic/elasticsearch/issues/16082), I am going to ask all relevant questions here from now on.
How can I turn off cluster routing allocation disk threshold? Second, if it is enabled and I am getting "high disk watermark exceeded on one or more nodes, rerouting shards" warning, will it stop logs being dumped on shared drive?
Now, I am also start getting "observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]", what is this?
---
### Author:  [@bleskes](https://discuss.elastic.co/u/bleskes)
#### Post date: [January 20, 2016, 1:55pm UTC](https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506/14 "2016-01-20T13:55:20Z")
> How can I turn off cluster routing allocation disk threshold?
Like so:
```
curl -XPUT "http://localhost:9200/_cluster/settings" -d'
{
"persistent": {
"cluster": {
"routing": {
"allocation.disk.threshold_enabled": false
}
}
}
}'
```
> [@thyfere](#):
>
> f it is enabled and I am getting "high disk watermark exceeded on one or more nodes, rerouting shards" warning, will it stop logs being dumped on shared drive?
Not sure what you mean to be honest. Which logs do you mean?
> [@thyfere](#):
>
> observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
I need to know where this comes from to say. What is the first part of that line?
---
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [January 21, 2016, 6:54am UTC](https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506/15 "2016-01-21T06:54:21Z")
> [@bleskes](#):
>
> curl -XPUT "[http://localhost:9200/\_cluster/settings](http://localhost:9200/_cluster/settings)" -d'
> {
> "persistent": {
> "cluster": {
> "routing": {
> "allocation.disk.threshold\_enabled": false
> }
> }
> }
> }'
When I run it, it throws the following error:
{"error":"JsonParseException[Unexpected character (''' (code 39)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')\n at [Source: [B@70a96f8c; line: 1, column: 2]]","status":500}curl: (6) Could not resolve host: persistentcurl: (3) [globbing] unmatched brace in column 1 curl: (6) Could not resolve host: cluster curl: (3) [globbing] unmatched brace in column 1 curl: (6) Could not resolve host: routing curl: (3) [globbing] unmatched brace in column 1 curl: (6) Could not resolve host: allocation.disk.threshold\_enabled curl: (6) Could not resolve host: false curl: (3) [globbing] unmatched close brace/bracket in column 1 curl: (3) [globbing] unmatched close brace/bracket in column 1 curl: (3) [globbing] unmatched close brace/bracket in column 1
---
### Author:  [@bleskes](https://discuss.elastic.co/u/bleskes)
#### Post date: [January 21, 2016, 7:40am UTC](https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506/16 "2016-01-21T07:40:08Z")
I think something went wrong with the copy paste from here... also, I forgot that you use windows. The command I gave you is for linux. You will have to call a PUT request to the url I specified (replacing the hostname and port if needed). The body of the request should the part between the curly braces.
---
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [January 21, 2016, 8:41am UTC](https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506/17 "2016-01-21T08:41:06Z")
So now I ran the following command:
curl -put localhost:9200/\_cluster/settings -d '{"persistent" : {"cluster.routing.allocation.disk.threshold\_enabled" : false}}' still I got this error:
{"error":"InvalidIndexNameException[[\_cluster] Invalid index name [_cluster], mu
st not start with '_']","status":400}
Not Found
## Not Found
* * *
HTTP Error 404. The requested resource is not found.
curl: (3) [globbing] unmatched brace in column 1 curl: (6) Could not resolve host: cluster.routing.allocation.disk.threshold\_enab led Not Found
## Not Found
* * *
HTTP Error 404. The requested resource is not found.
curl: (6) Could not resolve host: false curl: (3) [globbing] unmatched close brace/bracket in column 1 curl: (3) [globbing] unmatched close brace/bracket in column 1
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 11:22pm UTC](https://discuss.elastic.co/t/high-disk-watermark-exceeded-on-one-or-more-nodes-rerouting-shards/38506/18 "2017-07-05T23:22:40Z")