Elastic Watcher custom alert help

[sumitmadhale]

Hi,
We are using elastic cloud Watcher for custom alerting, what I wanted to achieve is to search a field "docker.image" for all the docker containers and check if it is changed since one minute, if it has changed take a action and send a slack notification with new docker.image field value. I have gone through Watcher documentation and getting started guides but I could not find a necessary information about setting a condition on each field that is returned as part of search. Can someone guide me if its possible with Watcher? Or any useful links for advanced usage.

[tsullivan (Tim Sullivan)]

There is a lot to unpack here as far as implementation details. At a high level, you need to:

1. Define the trigger time, for example: once per minute
2. Add a "search" input to get the sets of "now" images and "past" images
3. Add a "script" condition to get the sets from the search results, and compare the two sets. If there are any differences, return them
4. Add an "action" to send a message in Slack about the differences that were returned

I understand that it is hard to find all the info in Watcher docs, because implementing this presumes that you know how to write the query for the search input, and the Painless code for the script condition.

For the search, I would use a date histogram aggregation to bucket the data into per-minute groups. Then within those groups, I would get all the known images using a terms aggregation. I would add a range filter on the timestamp field to isolate the data between 1 minute ago and now.

I have data that collects country codes over time, and so if I wanted to get a group of "now" countries and "1 month ago" countries, my query looks like:

{
  "query": {
    "range": {
      "@date": { # (1)
        "gte": "now-1M/M",
        "lte": "now/M"
      }
    }
  },
  "aggs": {
    "minutes": {
      "date_histogram": { # (2)
        "field": "@date",
        "calendar_interval": "1M"
      },
      "aggs": {
        "countries": {
          "terms": { # (3)
            "field": "country",
            "size": 10
          }
        }
      }
    }
  }
}

• (1) isolates the data from one month ago to now, rounded down to the beginning of last month and up to beginning of this month
• (2) the "minutes" aggregation is per-month groups of the data
• (3) the "countries" aggregation is the top 10 country codes nested in the per-month groups

That query shows me results like:

{
  "aggregations" : {
    "minutes" : {
      "buckets" : [
        {
          "key_as_string" : "2020-10-01T00:00:00.000Z",
          "key" : 1601510400000,
          "doc_count" : 75,
          "countries" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 53,
            "buckets" : [
              {
                "key" : "BZ",
                "doc_count" : 3
              },
              {
                "key" : "PK",
                "doc_count" : 3
              },
              {
                "key" : "CW",
                "doc_count" : 2
              },
              {
                "key" : "HN",
                "doc_count" : 2
              },
              {
                "key" : "JE",
                "doc_count" : 2
              },
              {
                "key" : "PM",
                "doc_count" : 2
              },
              {
                "key" : "PW",
                "doc_count" : 2
              },
              {
                "key" : "SB",
                "doc_count" : 2
              },
              {
                "key" : "SG",
                "doc_count" : 2
              },
              {
                "key" : "SY",
                "doc_count" : 2
              }
            ]
          }
        },
        {
          "key_as_string" : "2020-11-01T00:00:00.000Z",
          "key" : 1604188800000,
          "doc_count" : 9,
          "countries" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "AL",
                "doc_count" : 1
              },
              {
                "key" : "AM",
                "doc_count" : 1
              },
              {
                "key" : "BE",
                "doc_count" : 1
              },
              {
                "key" : "BT",
                "doc_count" : 1
              },
              {
                "key" : "CF",
                "doc_count" : 1
              },
              {
                "key" : "GU",
                "doc_count" : 1
              },
              {
                "key" : "LK",
                "doc_count" : 1
              },
              {
                "key" : "NU",
                "doc_count" : 1
              },
              {
                "key" : "SM",
                "doc_count" : 1
              }
            ]
          }
        }
      ]
    }
  }
}

For the next part on writing the script condition to scan those results, I have to ask that you explore the Painless language on your own, or reach out for help in the Elasticsearch forums. I am not experienced enough in Painless to know how to write a script that does comparison of 2 sets .