How to set up alert based on value change over given amount of time

[Ruben_Bajo]

I have a costume log that contains a value that constantly increases over the day and gets back to 0 every midnight. I would like to set up a rule that can alert if this growth stops but I did not find a way to do that.

The problem is that the logs themselves are generated every few minutes regardless of how much the value increases so I can't monitor the log count, and I failed to find a method to compare two values with time shifted and set up a rule based on that difference.

[Wave (Andrew Sullivan)]

Hi @Ruben_Bajo,
Just a thought but aggregations might provide you the solution you need. For example if the last 10 values are all 10 the average value of those 10 values will be 10. If the last 10 values are from 1 to 10, the average will be 5.5. Or in other words if the latest value is greater than the average then you know it is increasing. If your latest value is exactly the same as the average then you know it hasn't increased within window you are averaging on. I hope I'm explaining that clearly.

You should be able to make a watcher that can run those aggregations and then throw an alert when those conditions are met.

[Ruben_Bajo]

Thank you! I got side tracked with other work but I will see if I can make it the way you said.

[Ruben_Bajo]

I did look through all the options when creating a rule but I can't find any, where I can compare two log values, or their averages.

[Wave (Andrew Sullivan)]

You're probably going to need to create an advance watcher, and then write a query with an aggregation. I can give you a an example if you like.

[Ruben_Bajo]

I would appreciate that very much

[Wave (Andrew Sullivan)]

Sorry for the delay. Consider the following:

POST /my-index-000001/_doc/?pretty
{
  "@timestamp": "2023-09-14T11:01:00",
  "value_a": 10
}

POST /my-index-000001/_doc/?pretty
{
  "@timestamp": "2023-09-14T11:02:00",
  "value_a": 10
}


POST /my-index-000001/_doc/?pretty
{
  "@timestamp": "2023-09-14T11:03:00",
  "value_a": 10
}

POST /my-index-000001/_doc/?pretty
{
  "@timestamp": "2023-09-14T11:04:00",
  "value_a": 10
}

POST /my-index-000001/_doc/?pretty
{
  "@timestamp": "2023-09-14T11:05:00",
  "value_a": 10
}

POST /my-index-000001/_doc/?pretty
{
  "@timestamp": "2023-09-14T11:06:00",
  "value_a": 10
}


POST /my-index-000001/_doc/?pretty
{
  "@timestamp": "2023-09-14T11:07:00",
  "value_a": 10
}

POST /my-index-000001/_doc/?pretty
{
  "@timestamp": "2023-09-14T11:08:00",
  "value_a": 11
}

POST /my-index-000001/_doc/?pretty
{
  "@timestamp": "2023-09-14T11:09:00",
  "value_a": 12
}

POST /my-index-000001/_doc/?pretty
{
  "@timestamp": "2023-09-14T11:10:00",
  "value_a": 13
}

You could look at the first five minutes with something like this:

GET /my-index-000001/_search
{
  "size":0,
  "aggs": {
    "average": {
      "avg": {
        "field": "value_a"
      }
    },
    "max_value": {
      "max": {
        "field": "value_a"
      }
    }
  },
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
              "gte": "2023-09-14T11:01:00",
              "lte": "2023-09-14T11:05:00"
            }
          }
        }
      ]
    }
  }
}

Which has the output of:

{
  "took": 4,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 5,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "average": {
      "value": 10
    },
    "max_value": {
      "value": 10
    }
  }
}

As you can see the average and max value are the same value so there has been no increase.
Switching up the five minute window to the last five minutes with:

GET /my-index-000001/_search
{
  "size":0,
  "aggs": {
    "average": {
      "avg": {
        "field": "value_a"
      }
    },
    "max_value": {
      "max": {
        "field": "value_a"
      }
    }
  },
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
              "gte": "2023-09-14T11:06:00",
              "lte": "2023-09-14T11:10:00"
            }
          }
        }
      ]
    }
  }
}

Gives us:

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 5,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "average": {
      "value": 11.2
    },
    "max_value": {
      "value": 13
    }
  }
}

The max value is greater than the average so you know that the value is increasing.

Hope this helps.