How would I create a watcher for disk space for my hosts? I am using the UI, but I think I will need to do a custom JSON watcher. I want to be notified when system.filesystem.used.pct is over a certain percentage for each of my hosts. Any tips?
How about starting with the query first and then have a proper watch. I think searching for documents from the last n minutes, where the pct is greater than the threshold and then aggregating on the hostname field (maybe sub aggregating on the mount point to have more details) sounds like a good start.
From there on, once the query returns what you need, you can go and create a watch out of that.
Does that make sense to you?
I selected Watcher and then selected Create Threshold Alert. You can see below what I see. Where would I aggregate such as you suggested? It sounds like you are suggesting not using this interface. How would I get to what you are talking about? Thank you for your help
You would need to write a watcher manually via the dev tools console or using the 'advanced watch' functionality in the watcher UI.
Okay thank you for your responses. I have now tried to use the advanced watch functionality in the watcher UI, and I copied and pasted the watch.json file from the filesystem_usage example in the github repo (https://github.com/elastic/examples/blob/master/Alerting/Sample%20Watches/filesystem_usage/watch.json). When I clicked Create Watch, it just kept processing for over 15 minutes before I killed it. Do you have any idea why it hung like that? Is that a normal processing time?
If there are any other examples of disk space alerts that notify with the exact hostname, please let me know.
What Elasticsearch version are you using and did you modify anything on that watch? I'll try to reproduce locally.
7.6 and no I did not. I just wanted to see if I could get it to work, and then I was planning on modifying it.
Thank you
can you be exact with the version, there is more than one 7.6 version. Thank you!
7.6.0
Hm, somehow the parsing of the watch is broken due to the watch being wrong.
See the last logging part
"actions": {
"log": {
"logging": {
"text": {
"inline": "foo"
}
}
}
needs to be
"actions": {
"log": {
"logging": {
"text": "foo"
}
}
I'll open an issue in kibana for that, as it should not hang nontheless
Thank you very much! That did not hang when I just ran it. However, I don't believe that I have it all set up for this watcher to work for me. Within the documentation, it says "This watch can be adapted to work with either topbeat or metricbeat data." How can I adapt it to metricbeat so that the following can be disregarded and it looks for metricbeat system.filesystem.used.pct?: "The watch assumes data is indexed into an index 'logs' with type 'filesystem'."?
I appreciate you working through this with me.
This will be fixed in the next versions of Kibana, see https://github.com/elastic/kibana/pull/60169
1 Like
Great thanks. However, I would like to configure that to be able to look at system.filesystem.used.pct from metricbeat instead. If you look at my comment from 3 days ago, I go into more depth. How would I accomplish this? Thanks
the indices and types field content needs to be adapted to your current data, as this is for an older major version (you can call GET _cat/indices to get an overview over your indices). The types can be removed, as types have been removed and deprecated in newer versions.
I highly encourage you to read https://www.elastic.co/blog/watching-the-watches-writing-debugging-and-testing-watches which is a lengthy but useful manually to get up and running with watch writing and debugging and then we can take it from there.
Thank you!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.