Parametrized watch

Elastic Stack Elasticsearch
1

I gather logs from 500 different devices (each separated with different id) and would like to create a watcher that detects when any of them stops sending data. I managed to create a watcher that does it for a single device and I put 500 watches using curl. But then I got an error that only limited number of watches can run at the same time.
I am trying to create a single watch and execute it (even externally) with different device IDs to check for lack of messages. Is that possible to do? I am trying to alter some data in my JSON watch, but without success. Is there any way to change metadata while executing watch?
Or can I use alternative input to change just part of the data instead of removing the whole predefined one?

2

You do not necessarily need one watch per host. If you instead create a chained watch similar to this example you can have one query that retrieves the IDs of all hosts that provided data e.g. within the last 1 hour and then compare this to a second query which retrieves a list of the hosts that provided data during the last x minutes. Then compare these sets to identify any host that has stopped sending data. This approach should scale and perform much better.

2 Likes
3

That's the elegant solution I was looking for :slight_smile:

I wonder if I could create a fixed list of IDs I expect to receive data from and compare it with those that actually send data. In my case I still didn't receive data from few devices and would like to detect them too.

4

You could create a separate index where you store a document per host you expect to receive from and use that in an additional step. That could also allow you to mark hosts down for maintenance etc.

5

I'm thinking of implementing something like this on my own.
But I have an idea to watch monitoring index. So I would be able to get if a beat is down, even if it hasn't had any data to send in the last 5 hours.

6

Hey @The,

maintaining a fixed list can be tedious but is a valid solution in some cases. Alternatively you could run the same query for two different time windows use a terms aggregation to get the ids found and then compare those two lists - if a id is missing from the newer time windows, you know the host has not sent any data.

--Alex

7

If you want to create a fixed list of IDs but not store them in an index .You can do it by using a stored script field.

    POST _scripts/dat_super_script
        {
          "script": {
            "lang": "painless",
            "code": """def list_of_my_IDS = ["ID1","ID2","ID3"]; 
     ---put some code here to compare theses IDs with the result found in the  payload
 and return true or false ---;
        """
            }
        }

And then in your watch , you have to use the stored script as conditon validator

  "email_administrator": {
  "condition": {
    "script": {
      "stored": "dat_super_script"
    }
  },
1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.