# Low disk watermark exceeded **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [March 3, 2019, 10:48pm UTC](https://discuss.elastic.co/t/low-disk-watermark-exceeded/170663 "2019-03-03T22:48:36Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![huzefabootwala](https://avatars.discourse-cdn.com/v4/letter/h/65b543/32.png) [@huzefabootwala](https://discuss.elastic.co/u/huzefabootwala)\ **Post date:** [March 3, 2019, 10:48pm UTC](https://discuss.elastic.co/t/low-disk-watermark-exceeded/170663/1 "2019-03-03T22:48:36Z")
Hello, I have installed Elasticsearch, Kibana and filebeat 6.5.4. I am planning to fetch Windows logs and logs from a custom directory. I have been successful in doing this but with some issues. Issue 1: [2019-03-04T09:40:29,629][INFO][o.e.c.r.a.DiskThresholdMonitor] [SERVERNAME] low disk watermark [85%] exceeded on [5lx750qwSgi6NyP0UXEQng][SERVERNAME][C:\ProgramData\Elastic\Elasticsearch\data\nodes\0] free: 26.3gb[11.3%], replicas will not be assigned to this node I need a solution for this. Have read other threads regarding this and have changed the config file setting low and high watermark but seems it is still an issue. Issue 2: I am not able to fetch logs from a custom directory in a correct manner. Logs do appear in kibana but only first few lines of the log file. Also, when kibana is running, size of my C drive keeps on decreasing. When I started kibana, available space in C drive was 32 Gb and after a while it dropped to 26 Gb. What is causing this? Any help would be appreciated. Thanx. ---
**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\ **Post date:** [March 6, 2019, 4:16pm UTC](https://discuss.elastic.co/t/low-disk-watermark-exceeded/170663/2 "2019-03-06T16:16:55Z")
Are you running the stack as a development environment and not a production environment? ---
**Author:** ![huzefabootwala](https://avatars.discourse-cdn.com/v4/letter/h/65b543/32.png) [@huzefabootwala](https://discuss.elastic.co/u/huzefabootwala)\ **Post date:** [March 6, 2019, 9:21pm UTC](https://discuss.elastic.co/t/low-disk-watermark-exceeded/170663/3 "2019-03-06T21:21:22Z")
Hi, I am setting up and testing the tool on my local machine first and then we would be using it for our production (logs management). Thanx. ---
**Author:** ![huzefabootwala](https://avatars.discourse-cdn.com/v4/letter/h/65b543/32.png) [@huzefabootwala](https://discuss.elastic.co/u/huzefabootwala)\ **Post date:** [March 6, 2019, 10:21pm UTC](https://discuss.elastic.co/t/low-disk-watermark-exceeded/170663/4 "2019-03-06T22:21:14Z")
How would I run the stack as a production? How different it would be than running the stack as development? Thanx. ---
**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\ **Post date:** [March 7, 2019, 2:34pm UTC](https://discuss.elastic.co/t/low-disk-watermark-exceeded/170663/5 "2019-03-07T14:34:40Z")
> [@huzefabootwala](#): > > Issue 1: [2019-03-04T09:40:29,629][INFO][o.e.c.r.a.DiskThresholdMonitor] [SERVERNAME] low disk watermark [85%] exceeded on [5lx750qwSgi6NyP0UXEQng][SERVERNAME][C:\ProgramData\Elastic\Elasticsearch\data\nodes\0] free: 26.3gb[11.3%], replicas will not be assigned to this node Elasticsearch uses conservative values to make sure it can correctly allocate replica of the shards, some operation on the shards require disk space, Elasticsearch uses these values as **guards** , but it's possible to change the threshold, you have to define the following in your `config/elasticsearch.yml` and restart it. > `cluster.routing.allocation.disk.watermark.low` > Controls the low watermark for disk usage. It defaults to `85%` , meaning that Elasticsearch will not allocate shards to nodes that have more than 85% disk used. It can also be set to an absolute byte value (like `500mb` ) to prevent Elasticsearch from allocating shards if less than the specified amount of space is available. This setting has no effect on the primary shards of newly-created indices or, specifically, any shards that have never previously been allocated. > Issue 2: I am not able to fetch logs from a custom directory in a correct manner. Logs do appear in kibana but only first few lines of the log file. For the above, I will need more details are you using LS or Filebeat if so can you share your configuration file and a bit of information about the environment? > Also, when kibana is running, size of my C drive keeps on decreasing. When I started kibana, available space in C drive was 32 Gb and after a while it dropped to 26 Gb. What is causing this? This might be better asked on the Kibana forum, but I think we should fix the log issue first. ---
**Author:** ![huzefabootwala](https://avatars.discourse-cdn.com/v4/letter/h/65b543/32.png) [@huzefabootwala](https://discuss.elastic.co/u/huzefabootwala)\ **Post date:** [March 7, 2019, 10:13pm UTC](https://discuss.elastic.co/t/low-disk-watermark-exceeded/170663/6 "2019-03-07T22:13:29Z")
Hi, thanx for your reply. **Below are the details of elasticsearch** { "name" : "...........", "cluster\_name" : "elasticsearch", "cluster\_uuid" : "rybM16R7TWOI7ZvY19gsHQ", "version" : { "number" : "6.5.4", "build\_flavor" : "default", "build\_type" : "zip", "build\_hash" : "d2ef93d", "build\_date" : "2018-12-17T21:17:40.758843Z", "build\_snapshot" : false, "lucene\_version" : "7.5.0", "minimum\_wire\_compatibility\_version" : "5.6.0", "minimum\_index\_compatibility\_version" : "5.0.0" }, "tagline" : "You Know, for Search" } **I have set the following values in elasticsearch.yml** cluster.name: elasticsearch cluster.routing.allocation.disk.threshold\_enabled: true cluster.routing.allocation.disk.watermark.flood\_stage: 5gb cluster.routing.allocation.disk.watermark.low: 20gb cluster.routing.allocation.disk.watermark.high: 15gb **Still I am getting the same error of disk watermark.** **I am using Filebeat to fetch Windows logs and logs from a custom directory.** Filebeat.yml #==================== Elasticsearch template setting ========================== setup.template.settings: index.number\_of\_shards: 3 #index.codec: best\_compression #\_source.enabled: false filebeat.inputs: - type: log I have just stored a single file in Test folder and I can see logs in Kibana. **I have the following concerns:** We need to insert around 5 Gb of data each day. We need to retain logs for 365 days and perform active search on it. Will this configuration be appropriate for the above scenario? Thanx. ---
**Author:** ![huzefabootwala](https://avatars.discourse-cdn.com/v4/letter/h/65b543/32.png) [@huzefabootwala](https://discuss.elastic.co/u/huzefabootwala)\ **Post date:** [April 2, 2019, 9:37pm UTC](https://discuss.elastic.co/t/low-disk-watermark-exceeded/170663/7 "2019-04-02T21:37:13Z")
Hello, Any updates on this please? Thanx. ---
**Author:** ![staodd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/staodd/32/24509_2.png) [@staodd](https://discuss.elastic.co/u/staodd)\ **Post date:** [April 2, 2019, 10:06pm UTC](https://discuss.elastic.co/t/low-disk-watermark-exceeded/170663/8 "2019-04-02T22:06:20Z")
What setup would be appropriate for what? You don't say anything about the hardware you plan to run in production. So how can anyone make a qualified statement if it's enough? ---
**Author:** ![huzefabootwala](https://avatars.discourse-cdn.com/v4/letter/h/65b543/32.png) [@huzefabootwala](https://discuss.elastic.co/u/huzefabootwala)\ **Post date:** [April 3, 2019, 4:29am UTC](https://discuss.elastic.co/t/low-disk-watermark-exceeded/170663/9 "2019-04-03T04:29:37Z")
Hi, thanx for the reply. I figured what the issue was. Sorry about the confusion. I have set up the elastic search on my local machine but the prod machine would be a server probably 1 TB. Thank you. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 1, 2019, 4:29am UTC](https://discuss.elastic.co/t/low-disk-watermark-exceeded/170663/10 "2019-05-01T04:29:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.