# Watcher for Disk Space

**URL:** <https://discuss.elastic.co/t/watcher-for-disk-space/222817>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-alerting\
**Created:** [March 9, 2020, 10:34pm UTC](https://discuss.elastic.co/t/watcher-for-disk-space/222817 "2020-03-09T22:34:56Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![sroseman](https://avatars.discourse-cdn.com/v4/letter/s/b4bc9f/32.png) [@sroseman](https://discuss.elastic.co/u/sroseman)\
**Post date:** [March 9, 2020, 10:34pm UTC](https://discuss.elastic.co/t/watcher-for-disk-space/222817/1 "2020-03-09T22:34:56Z")

</div>

How would I create a watcher for disk space for my hosts? I am using the UI, but I think I will need to do a custom JSON watcher. I want to be notified when system.filesystem.used.pct is over a certain percentage for each of my hosts. Any tips?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [March 10, 2020, 1:24pm UTC](https://discuss.elastic.co/t/watcher-for-disk-space/222817/2 "2020-03-10T13:24:10Z")

</div>

How about starting with the query first and then have a proper watch. I think searching for documents from the last `n` minutes, where the `pct` is greater than the threshold and then aggregating on the hostname field (maybe sub aggregating on the mount point to have more details) sounds like a good start.

From there on, once the query returns what you need, you can go and create a watch out of that.

Does that make sense to you?

---

<div class="post-metadata">

**Author:** ![sroseman](https://avatars.discourse-cdn.com/v4/letter/s/b4bc9f/32.png) [@sroseman](https://discuss.elastic.co/u/sroseman)\
**Post date:** [March 10, 2020, 3:33pm UTC](https://discuss.elastic.co/t/watcher-for-disk-space/222817/3 "2020-03-10T15:33:05Z")

</div>

I selected Watcher and then selected Create Threshold Alert. You can see below what I see. Where would I aggregate such as you suggested? It sounds like you are suggesting not using this interface. How would I get to what you are talking about? Thank you for your help

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/f/5f3715fe3a25ad99b91fd30801d7433675959c31.png)

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [March 12, 2020, 10:07am UTC](https://discuss.elastic.co/t/watcher-for-disk-space/222817/4 "2020-03-12T10:07:32Z")

</div>

You would need to write a watcher manually via the dev tools console or using the 'advanced watch' functionality in the watcher UI.

---

<div class="post-metadata">

**Author:** ![sroseman](https://avatars.discourse-cdn.com/v4/letter/s/b4bc9f/32.png) [@sroseman](https://discuss.elastic.co/u/sroseman)\
**Post date:** [March 12, 2020, 1:48pm UTC](https://discuss.elastic.co/t/watcher-for-disk-space/222817/5 "2020-03-12T13:48:46Z")

</div>

Okay thank you for your responses. I have now tried to use the advanced watch functionality in the watcher UI, and I copied and pasted the watch.json file from the filesystem\_usage example in the github repo ([https://github.com/elastic/examples/blob/master/Alerting/Sample%20Watches/filesystem\_usage/watch.json](https://github.com/elastic/examples/blob/master/Alerting/Sample%20Watches/filesystem_usage/watch.json)). When I clicked Create Watch, it just kept processing for over 15 minutes before I killed it. Do you have any idea why it hung like that? Is that a normal processing time?

If there are any other examples of disk space alerts that notify with the exact hostname, please let me know.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [March 12, 2020, 4:07pm UTC](https://discuss.elastic.co/t/watcher-for-disk-space/222817/6 "2020-03-12T16:07:06Z")

</div>

What Elasticsearch version are you using and did you modify anything on that watch? I'll try to reproduce locally.

---

<div class="post-metadata">

**Author:** ![sroseman](https://avatars.discourse-cdn.com/v4/letter/s/b4bc9f/32.png) [@sroseman](https://discuss.elastic.co/u/sroseman)\
**Post date:** [March 13, 2020, 1:12am UTC](https://discuss.elastic.co/t/watcher-for-disk-space/222817/7 "2020-03-13T01:12:11Z")

</div>

7.6 and no I did not. I just wanted to see if I could get it to work, and then I was planning on modifying it.  
Thank you

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [March 13, 2020, 9:14am UTC](https://discuss.elastic.co/t/watcher-for-disk-space/222817/8 "2020-03-13T09:14:31Z")

</div>

can you be exact with the version, there is more than one 7.6 version. Thank you!

---

<div class="post-metadata">

**Author:** ![sroseman](https://avatars.discourse-cdn.com/v4/letter/s/b4bc9f/32.png) [@sroseman](https://discuss.elastic.co/u/sroseman)\
**Post date:** [March 13, 2020, 1:59pm UTC](https://discuss.elastic.co/t/watcher-for-disk-space/222817/9 "2020-03-13T13:59:47Z")

</div>

7.6.0

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [March 13, 2020, 5:01pm UTC](https://discuss.elastic.co/t/watcher-for-disk-space/222817/10 "2020-03-13T17:01:57Z")

</div>

Hm, somehow the parsing of the watch is broken due to the watch being wrong.

See the last logging part

```auto
  "actions": {
    "log": {
      "logging": {
        "text": {
          "inline": "foo"
        }
      }
    }

```

needs to be

```auto
  "actions": {
    "log": {
      "logging": {
        "text": "foo"
      }
    }

```

I'll open an issue in kibana for that, as it should not hang nontheless

---

<div class="post-metadata">

**Author:** ![sroseman](https://avatars.discourse-cdn.com/v4/letter/s/b4bc9f/32.png) [@sroseman](https://discuss.elastic.co/u/sroseman)\
**Post date:** [March 13, 2020, 10:31pm UTC](https://discuss.elastic.co/t/watcher-for-disk-space/222817/11 "2020-03-13T22:31:22Z")

</div>

Thank you very much! That did not hang when I just ran it. However, I don't believe that I have it all set up for this watcher to work for me. Within the documentation, it says "This watch can be adapted to work with either topbeat or metricbeat data." How can I adapt it to metricbeat so that the following can be disregarded and it looks for metricbeat system.filesystem.used.pct?: "The watch assumes data is indexed into an index 'logs' with type 'filesystem'."?  
I appreciate you working through this with me.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [March 15, 2020, 6:53pm UTC](https://discuss.elastic.co/t/watcher-for-disk-space/222817/12 "2020-03-15T18:53:34Z")

</div>

This will be fixed in the next versions of Kibana, see [https://github.com/elastic/kibana/pull/60169](https://github.com/elastic/kibana/pull/60169)

---

<div class="post-metadata">

**Author:** ![sroseman](https://avatars.discourse-cdn.com/v4/letter/s/b4bc9f/32.png) [@sroseman](https://discuss.elastic.co/u/sroseman)\
**Post date:** [March 16, 2020, 6:10pm UTC](https://discuss.elastic.co/t/watcher-for-disk-space/222817/13 "2020-03-16T18:10:44Z")

</div>

Great thanks. However, I would like to configure that to be able to look at system.filesystem.used.pct from metricbeat instead. If you look at my comment from 3 days ago, I go into more depth. How would I accomplish this? Thanks

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [March 17, 2020, 1:49pm UTC](https://discuss.elastic.co/t/watcher-for-disk-space/222817/14 "2020-03-17T13:49:55Z")

</div>

the `indices` and `types` field content needs to be adapted to your current data, as this is for an older major version (you can call `GET _cat/indices` to get an overview over your indices). The `types` can be removed, as types have been removed and deprecated in newer versions.

I highly encourage you to read [https://www.elastic.co/blog/watching-the-watches-writing-debugging-and-testing-watches](https://www.elastic.co/blog/watching-the-watches-writing-debugging-and-testing-watches) which is a lengthy but useful manually to get up and running with watch writing and debugging and then we can take it from there.

---

<div class="post-metadata">

**Author:** ![sroseman](https://avatars.discourse-cdn.com/v4/letter/s/b4bc9f/32.png) [@sroseman](https://discuss.elastic.co/u/sroseman)\
**Post date:** [March 17, 2020, 3:31pm UTC](https://discuss.elastic.co/t/watcher-for-disk-space/222817/15 "2020-03-17T15:31:18Z")

</div>

Thank you!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 14, 2020, 3:31pm UTC](https://discuss.elastic.co/t/watcher-for-disk-space/222817/16 "2020-04-14T15:31:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
