Parametrized watch

I gather logs from 500 different devices (each separated with different id) and would like to create a watcher that detects when any of them stops sending data. I managed to create a watcher that does it for a single device and I put 500 watches using curl. But then I got an error that only limited number of watches can run at the same time.
I am trying to create a single watch and execute it (even externally) with different device IDs to check for lack of messages. Is that possible to do? I am trying to alter some data in my JSON watch, but without success. Is there any way to change metadata while executing watch?
Or can I use alternative input to change just part of the data instead of removing the whole predefined one?

You do not necessarily need one watch per host. If you instead create a chained watch similar to this example you can have one query that retrieves the IDs of all hosts that provided data e.g. within the last 1 hour and then compare this to a second query which retrieves a list of the hosts that provided data during the last x minutes. Then compare these sets to identify any host that has stopped sending data. This approach should scale and perform much better.

2 Likes

That's the elegant solution I was looking for :slight_smile:

I wonder if I could create a fixed list of IDs I expect to receive data from and compare it with those that actually send data. In my case I still didn't receive data from few devices and would like to detect them too.

You could create a separate index where you store a document per host you expect to receive from and use that in an additional step. That could also allow you to mark hosts down for maintenance etc.

I'm thinking of implementing something like this on my own.
But I have an idea to watch monitoring index. So I would be able to get if a beat is down, even if it hasn't had any data to send in the last 5 hours.

Hey @The,

maintaining a fixed list can be tedious but is a valid solution in some cases. Alternatively you could run the same query for two different time windows use a terms aggregation to get the ids found and then compare those two lists - if a id is missing from the newer time windows, you know the host has not sent any data.

--Alex

If you want to create a fixed list of IDs but not store them in an index .You can do it by using a stored script field.

    POST _scripts/dat_super_script
        {
          "script": {
            "lang": "painless",
            "code": """def list_of_my_IDS = ["ID1","ID2","ID3"]; 
     ---put some code here to compare theses IDs with the result found in the  payload
 and return true or false ---;
        """
            }
        }

And then in your watch , you have to use the stored script as conditon validator

  "email_administrator": {
  "condition": {
    "script": {
      "stored": "dat_super_script"
    }
  },
1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.