Alerting on derivative

[torbjornkarlsson]

Hi.

It is fairly easy to create a visualization that shows spikes in number of requests (using derivative). But how do I go about to create a watcher that triggers on a big increase in number of requests?

Ideally, we would like to trigger alerts based on a Timelion calculation. Is this supported?

Some more details:

Our application is exposing metrics in the Prometheus format, this metric is collected with metricbeat and send to Elasticsearch.

The data for a specific http endpoint looks like this:

{
  "name": "http.server.requests",
  "measurements": [
    {
      "statistic": "COUNT",
      "value": 972.0
    },
    {
      "statistic": "TOTAL_TIME",
      "value": 276.310957666
    },
    {
      "statistic": "MAX",
      "value": 0.235733787
    }
  ],
  "availableTags": [
    {
      "tag": "exception",
      "values": [
        "None"
      ]
    },
    {
      "tag": "method",
      "values": [
        "GET"
      ]
    },
    {
      "tag": "status",
      "values": [
        "200"
      ]
    }
  ]
}

Where 972 is the number of hits on this endpoint since the application started, this value is always increasing. We also get the total execution time and the average time for requests.

Thanks in advance!

/Torbjörn

[spinscale (Alexander Reelsen)]

Please take the time to properly format JSON. You can use markdown in here and it eases the ability to read a post endlessly. Thanks a lot!

timelion calculation is not supported. What is the alerting condition you would like to trigger on? If the difference between two documents exceeds a certain threshold? Can you write a search that is doing this already or not? The latter question is important because if you are able to, it will be easy to write a watch.

--Alex

[torbjornkarlsson]

Hi.

Sorry for the missing formatting.

Yes, the condition would be that the difference between two documents exceed a threshold. And I currently don't know how to write that query, is there an example online I can look at?

Thanks for the reply,

/Torbjörn

[spinscale (Alexander Reelsen)]

comparing two documents directly is hard to do in elasticsearch unless you know their direct ids.

What you could do however is run an aggregation, that filters between two time ranges (i.e. now-5m and -5m-10m), then gets the max value using a max aggregation, and the watch condition compares those two values in the condition to decide if it should be triggered.

There are few alerting examples in the examples repo, but I do not know if something matches this requirement. You should find a few examples using aggregations though in there.

Hope that makes sense.

[torbjornkarlsson]

Ok, thanks.

I guess I have to learn a bit about aggregations...

Tanks for the reply!

/Torbjörn

[jmccain]

I use some alerts on derivatives similar to what Alex is describing that work well.

You might run into an issue due to the shape of your document, however. The "measurements" list of objects:

{
    "measurements": [
        {
        "statistic": "COUNT",
        "value": 972.0
        },
        {
        "statistic": "TOTAL_TIME",
        "value": 276.310957666
        },
        {
        "statistic": "MAX",
        "value": 0.235733787
        }
}

will be internally stored like this:

{
    "measurements.statistic": [
        "COUNT",
        "TOTAL_TIME",
        "MAX"
        ],
    "measurements.value": [
        972.0,
        276.310957666,
        0.235733787
    ]
}

So you will lose the association between each statistic and its value, making the kind of analysis you are looking for impossible.

To fix that, you have 2 options:
- Use a Nested datatype for measurements
- Use a Logstash filter to transform the data from a list of objects to nested properties (you can do this with the Ruby filter) like:

{
    "measurements.COUNT": 972.0,
    "measurements.TOTAL_TIME": 276.310957666,
    ...
}