ELK Watcher Response correct?

Elastic Stack Elasticsearch
1 
{
  "_id" : "metric_DashOperations_getUser_success_latency_8d38ec7e-62f1-4b53-8e01-baacab7dad47-2019-08-13T08:57:38.355Z",
  "watch_record" : {
    "watch_id" : "metric_DashOperations_getUser_success_latency",
    "node" : "1__-4RYsSCqVXSZna6cW4w",
    "state" : "execution_not_needed",
    "user" : "elastic",
    "status" : {
      "state" : {
        "active" : true,
        "timestamp" : "2019-08-13T06:33:16.756Z"
      },
      "last_checked" : "2019-08-13T08:57:38.355Z",
      "actions" : {
        "pagerduty" : {
          "ack" : {
            "timestamp" : "2019-08-13T06:33:16.756Z",
            "state" : "awaits_successful_execution"
          }
        }
      },
      "execution_state" : "execution_not_needed",
      "version" : 29
    },
    "trigger_event" : {
      "type" : "manual",
      "triggered_time" : "2019-08-13T08:57:38.352Z",
      "manual" : {
        "schedule" : {
          "scheduled_time" : "2019-08-13T08:57:38.352Z"
        }
      }
    },
    "input" : {
      "search" : {
        "request" : {
          "search_type" : "query_then_fetch",
          "indices" : [
            "prod-dash-*"
          ],
          "types" : [ ],
          "body" : {
            "size" : 0,
            "query" : {
              "bool" : {
                "filter" : {
                  "range" : {
                    "@timestamp" : {
                      "gte" : "{{ctx.trigger.scheduled_time}}||-3m",
                      "lte" : "{{ctx.trigger.scheduled_time}}",
                      "format" : "strict_date_optional_time||epoch_millis"
                    }
                  }
                }
              }
            },
            "aggs" : {
              "bucketAgg" : {
                "terms" : {
                  "field" : "metric_DashOperations_getUser_success_latency",
                  "size" : 3,
                  "order" : {
                    "metricAgg" : "desc"
                  }
                },
                "aggs" : {
                  "metricAgg" : {
                    "avg" : {
                      "field" : "metric_DashOperations_getUser_success_latency"
                    }
                  }
                }
              }
            }
          }
        }
      }
    },
    "condition" : {
      "script" : {
        "source" : "ArrayList arr = ctx.payload.aggregations.bucketAgg.buckets; for (int i = 0; i < arr.length; i++) { if (arr[i].doc_count > params.threshold) { return true; } } return false;",
        "lang" : "painless",
        "params" : {
          "threshold" : 10000
        }
      }
    },
    "metadata" : {
      "name" : "metric_DashOperations_getUser_success_latency",
      "xpack" : {
        "type" : "json"
      }
    },
    "result" : {
      "execution_time" : "2019-08-13T08:57:38.355Z",
      "execution_duration" : 110,
      "input" : {
        "type" : "search",
        "status" : "success",
        "payload" : {
          "_shards" : {
            "total" : 20,
            "failed" : 0,
            "successful" : 20,
            "skipped" : 0
          },
          "hits" : {
            "hits" : [ ],
            "total" : 0,
            "max_score" : 0.0
          },
          "took" : 4,
          "timed_out" : false,
          "aggregations" : {
            "bucketAgg" : {
              "doc_count_error_upper_bound" : 0,
              "sum_other_doc_count" : 0,
              "buckets" : [ ]
            }
          }
        },
        "search" : {
          "request" : {
            "search_type" : "query_then_fetch",
            "indices" : [
              "prod-dash-*"
            ],
            "types" : [ ],
            "body" : {
              "size" : 0,
              "query" : {
                "bool" : {
                  "filter" : {
                    "range" : {
                      "@timestamp" : {
                        "gte" : "2019-08-13T08:57:38.352Z||-3m",
                        "lte" : "2019-08-13T08:57:38.352Z",
                        "format" : "strict_date_optional_time||epoch_millis"
                      }
                    }
                  }
                }
              },
              "aggs" : {
                "bucketAgg" : {
                  "terms" : {
                    "field" : "metric_DashOperations_getUser_success_latency",
                    "size" : 3,
                    "order" : {
                      "metricAgg" : "desc"
                    }
                  },
                  "aggs" : {
                    "metricAgg" : {
                      "avg" : {
                        "field" : "metric_DashOperations_getUser_success_latency"
                      }
                    }
                  }
                }
              }
            }
          }
        }
      },
      "condition" : {
        "type" : "script",
        "status" : "success",
        "met" : false
      },
      "actions" : [ ]
    },
    "messages" : [ ]
  }
}

i used this command - POST _xpack/watcher/watch/metric_DashOperations_getUser_success_latency/_execute

Is the watcher working correctly or not? why my actions and messages field in result are blank?
@Badger

2

The execution is correct. There are no actions being executed, because the condition was not true. See the result.condition.met field

3

Actually i was looking to simulate this alarm without the condition being met so as to check if notifications are being triggered on pagerduty. is it possible to simulate it? @spinscale

4

Hey,

yes, check the options in the Execute Watch API. You can ignore the condition and execute anway or set an alternative_input. Also you can only simulate the execution of the actions setting the action_mode parameter.

--Alex

5

My main agenda is to calculate the average of top 5 datapoints in last 5 min and trigger a pagerduty action. i have written a following script for this. As per the output i am not sure if the execution is taking place as per the demands.
I am attaching the script please have a look @spinscale

 {
          "trigger": {
            "schedule": {
              "interval": "5m"
            }
          },
          "input" : {
              "search" : {
                "request" : {
                  "search_type" : "query_then_fetch",
                  "indices" : [
                    "geo*"
                  ],
                  "types" : [ ],
                  "body" : {
                    "size" : 0,
                    "query" : {
                      "bool" : {
                        "filter" : {
                          "range" : {
                            "@timestamp" : {
                              "gte" : "{{ctx.trigger.scheduled_time}}||-5m",
                              "lte" : "{{ctx.trigger.scheduled_time}}",
                              "format" : "strict_date_optional_time||epoch_millis"
                            }
                          }
                        }
                      }
                    },
                    "aggs" : {
                      "bucketAgg" : {
                        "terms" : {
                          "field" : "metric_GeocoderApi_success_latency",
                          "size" : 5,
                          "order" : {
                            "metricAgg" : "desc"
                          }
                        },
                        "aggs" : {
                          "metricAgg" : {
                            "avg" : {
                              "field" : "metric_GeocoderApi_success_latency"
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            },
            "condition" : {
              "script" : {
                "source" : "ArrayList arr = ctx.payload.aggregations.bucketAgg.buckets; for (int i = 0; i < arr.length; i++) { if (arr[i]['metricAgg'].value > params.threshold) { return true; } } return false;",
                "lang" : "painless",
                "params" : {
                  "threshold" : 60000
                }
              }
            },
          "actions": {
            "pagerduty": {
              "webhook": {
                "scheme": "https",
                "host": "events.pagerduty.com",
                "port": 443,
                "method": "post",
                "path": "/generic/create_event.json",
                "params": {},
                "headers": {
                  "Content-type": "application/json"
                },
                "body": "{\"service_key\": \"xxxxxxxxxxxxx\",\"event_type\": \"trigger\",\"description\": \"Issue in GeocoderApi_success_latency:GeocoderApi_success_latency>60000 in last 5 minutes\"}"
              }
            }
           }
        }
6

What exactly is your question here? Using the execute watch api you can simulate everything as needed and thus figure out if the execution has happened by specifying an alternative input. You need to check the output of the execute watch API to see what has happened.

Please take some time to read https://www.elastic.co/blog/watching-the-watches-writing-debugging-and-testing-watches - this is a great resource to learn to debug watches a more efficient.

If you have questions, please ask concrete questions after reviewing the execute watch API output instead of 'can you check this watch'. This will ease debugging a lot. Thank you!

7

@spinscale Let me Reframe my question.
What i expect- I have a metric say X, i wish to calculate the average of the metric X for last 5 min of its top 5 datapoints(from last 5 min ordered in descending form) and set the alarm if the average of these 5 datapoints meet the threshold.
In the comment above i have pasted the Advance watch script i created, i went through the doc you sent me.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.