Multi Metric Watcher won't trigger Action even though condition is met

Elastic Stack Kibana
,
1

Hello guys,

Hope you are all doing great - need some assistance on this issue i am experiencing where when i try to Simulate this watcher - it won't trigger the "action" it just leaves action blank as if conditions weren't met - when they are met.

{
  "watch_id": "_inlined_",
  "node": "O5vw_C5lTvq_VZ8EJwV-4g",
  "state": "execution_not_needed",
  "user": "User_ID",
  "status": {
    "state": {
      "active": true,
      "timestamp": "2021-06-21T04:11:57.130Z"
    },
    "last_checked": "2021-06-21T04:11:57.134Z",
    "actions": {
      "Watcher-webhook": {
        "ack": {
          "timestamp": "2021-06-21T04:11:57.130Z",
          "state": "awaits_successful_execution"
        }
      }
    },
    "execution_state": "execution_not_needed",
    "version": -1
  },
  "trigger_event": {
    "type": "manual",
    "triggered_time": "2021-06-21T04:11:57.131Z",
    "manual": {
      "schedule": {
        "scheduled_time": "2021-06-21T04:11:57.131Z"
      }
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          ".ml-anomalies-*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "filter": [
                {
                  "term": {
                    "job_id": "all-errors"
                  }
                },
                {
                  "range": {
                    "timestamp": {
                      "gte": "now-30m"
                    }
                  }
                },
                {
                  "terms": {
                    "result_type": [
                      "bucket",
                      "record",
                      "influencer"
                    ]
                  }
                }
              ]
            }
          },
          "aggs": {
            "bucket_results": {
              "filter": {
                "range": {
                  "anomaly_score": {
                    "gte": 0
                  }
                }
              },
              "aggs": {
                "top_bucket_hits": {
                  "top_hits": {
                    "sort": [
                      {
                        "anomaly_score": {
                          "order": "desc"
                        }
                      }
                    ],
                    "_source": {
                      "includes": [
                        "job_id",
                        "result_type",
                        "timestamp",
                        "anomaly_score",
                        "is_interim"
                      ]
                    },
                    "size": 1,
                    "script_fields": {
                      "start": {
                        "script": {
                          "lang": "painless",
                          "source": "LocalDateTime.ofEpochSecond((doc[\"timestamp\"].value.getMillis()-((doc[\"bucket_span\"].value * 1000)\n * params.padding)) / 1000, 0, ZoneOffset.UTC).toString()+\":00.000Z\"",
                          "params": {
                            "padding": 10
                          }
                        }
                      },
                      "end": {
                        "script": {
                          "lang": "painless",
                          "source": "LocalDateTime.ofEpochSecond((doc[\"timestamp\"].value.getMillis()+((doc[\"bucket_span\"].value * 1000)\n * params.padding)) / 1000, 0, ZoneOffset.UTC).toString()+\":00.000Z\"",
                          "params": {
                            "padding": 10
                          }
                        }
                      },
                      "timestamp_epoch": {
                        "script": {
                          "lang": "painless",
                          "source": "doc[\"timestamp\"].value.getMillis()/1000"
                        }
                      },
                      "timestamp_iso8601": {
                        "script": {
                          "lang": "painless",
                          "source": "doc[\"timestamp\"].value"
                        }
                      },
                      "score": {
                        "script": {
                          "lang": "painless",
                          "source": "Math.round(doc[\"anomaly_score\"].value)"
                        }
                      }
                    }
                  }
                }
              }
            },
            "influencer_results": {
              "filter": {
                "range": {
                  "influencer_score": {
                    "gte": 3
                  }
                }
              },
              "aggs": {
                "top_influencer_hits": {
                  "top_hits": {
                    "sort": [
                      {
                        "influencer_score": {
                          "order": "desc"
                        }
                      }
                    ],
                    "_source": {
                      "includes": [
                        "result_type",
                        "timestamp",
                        "influencer_field_name",
                        "influencer_field_value",
                        "influencer_score",
                        "isInterim"
                      ]
                    },
                    "size": 3,
                    "script_fields": {
                      "score": {
                        "script": {
                          "lang": "painless",
                          "source": "Math.round(doc[\"influencer_score\"].value)"
                        }
                      }
                    }
                  }
                }
              }
            },
            "record_results": {
              "filter": {
                "range": {
                  "record_score": {
                    "gte": 3
                  }
                }
              },
              "aggs": {
                "top_record_hits": {
                  "top_hits": {
                    "sort": [
                      {
                        "record_score": {
                          "order": "desc"
                        }
                      }
                    ],
                    "_source": {
                      "includes": [
                        "result_type",
                        "timestamp",
                        "record_score",
                        "is_interim",
                        "function",
                        "field_name",
                        "by_field_value",
                        "over_field_value",
                        "partition_field_value"
                      ]
                    },
                    "size": 3,
                    "script_fields": {
                      "score": {
                        "script": {
                          "lang": "painless",
                          "source": "Math.round(doc[\"record_score\"].value)"
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  },
2 
  "condition": {
    "compare": {
      "ctx.payload.aggregations.bucket_results.doc_count": {
        "gt": 0
      }
    }
  },
  "metadata": {
    "xpack": {
      "type": "json"
    }
  },
  "result": {
    "execution_time": "2021-06-21T04:11:57.134Z",
    "execution_duration": 17,
    "input": {
      "type": "search",
      "status": "success",
      "payload": {
        "_shards": {
          "total": 2,
          "failed": 0,
          "successful": 2,
          "skipped": 0
        },
        "hits": {
          "hits": [],
          "total": 0,
          "max_score": null
        },
        "took": 14,
        "timed_out": false,
        "aggregations": {
          "influencer_results": {
            "doc_count": 0,
            "top_influencer_hits": {
              "hits": {
                "hits": [],
                "total": 0,
                "max_score": null
              }
            }
          },
          "record_results": {
            "doc_count": 0,
            "top_record_hits": {
              "hits": {
                "hits": [],
                "total": 0,
                "max_score": null
              }
            }
          },
          "bucket_results": {
            "top_bucket_hits": {
              "hits": {
                "hits": [],
                "total": 0,
                "max_score": null
              }
            },
            "doc_count": 0
          }
        }
      },
      "search": {
        "request": {
          "search_type": "query_then_fetch",
          "indices": [
            ".ml-anomalies-*"
          ],
          "rest_total_hits_as_int": true,
          "body": {
            "size": 0,
            "query": {
              "bool": {
                "filter": [
                  {
                    "term": {
                      "job_id": "all-errors"
                    }
                  },
                  {
                    "range": {
                      "timestamp": {
                        "gte": "now-30m"
                      }
                    }
                  },
                  {
                    "terms": {
                      "result_type": [
                        "bucket",
                        "record",
                        "influencer"
                      ]
                    }
                  }
                ]
              }
            },
            "aggs": {
              "bucket_results": {
                "filter": {
                  "range": {
                    "anomaly_score": {
                      "gte": 0
                    }
                  }
                },
                "aggs": {
                  "top_bucket_hits": {
                    "top_hits": {
                      "sort": [
                        {
                          "anomaly_score": {
                            "order": "desc"
                          }
                        }
                      ],
                      "_source": {
                        "includes": [
                          "job_id",
                          "result_type",
                          "timestamp",
                          "anomaly_score",
                          "is_interim"
                        ]
                      },
                      "size": 1,
                      "script_fields": {
                        "start": {
                          "script": {
                            "lang": "painless",
                            "source": "LocalDateTime.ofEpochSecond((doc[\"timestamp\"].value.getMillis()-((doc[\"bucket_span\"].value * 1000)\n * params.padding)) / 1000, 0, ZoneOffset.UTC).toString()+\":00.000Z\"",
                            "params": {
                              "padding": 10
                            }
                          }
                        },
                        "end": {
                          "script": {
                            "lang": "painless",
                            "source": "LocalDateTime.ofEpochSecond((doc[\"timestamp\"].value.getMillis()+((doc[\"bucket_span\"].value * 1000)\n * params.padding)) / 1000, 0, ZoneOffset.UTC).toString()+\":00.000Z\"",
                            "params": {
                              "padding": 10
                            }
                          }
                        },
                        "timestamp_epoch": {
                          "script": {
                            "lang": "painless",
                            "source": "doc[\"timestamp\"].value.getMillis()/1000"
                          }
                        },
                        "timestamp_iso8601": {
                          "script": {
                            "lang": "painless",
                            "source": "doc[\"timestamp\"].value"
                          }
                        },
                        "score": {
                          "script": {
                            "lang": "painless",
                            "source": "Math.round(doc[\"anomaly_score\"].value)"
                          }
                        }
                      }
                    }
                  }
                }
              },
              "influencer_results": {
                "filter": {
                  "range": {
                    "influencer_score": {
                      "gte": 3
                    }
                  }
                },
                "aggs": {
                  "top_influencer_hits": {
                    "top_hits": {
                      "sort": [
                        {
                          "influencer_score": {
                            "order": "desc"
                          }
                        }
                      ],
                      "_source": {
                        "includes": [
                          "result_type",
                          "timestamp",
                          "influencer_field_name",
                          "influencer_field_value",
                          "influencer_score",
                          "isInterim"
                        ]
                      },
                      "size": 3,
                      "script_fields": {
                        "score": {
                          "script": {
                            "lang": "painless",
                            "source": "Math.round(doc[\"influencer_score\"].value)"
                          }
                        }
                      }
                    }
                  }
                }
              },
              "record_results": {
                "filter": {
                  "range": {
                    "record_score": {
                      "gte": 3
                    }
                  }
                },
                "aggs": {
                  "top_record_hits": {
                    "top_hits": {
                      "sort": [
                        {
                          "record_score": {
                            "order": "desc"
                          }
                        }
                      ],
                      "_source": {
                        "includes": [
                          "result_type",
                          "timestamp",
                          "record_score",
                          "is_interim",
                          "function",
                          "field_name",
                          "by_field_value",
                          "over_field_value",
                          "partition_field_value"
                        ]
                      },
                      "size": 3,
                      "script_fields": {
                        "score": {
                          "script": {
                            "lang": "painless",
                            "source": "Math.round(doc[\"record_score\"].value)"
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    },
    "condition": {
      "type": "compare",
      "status": "success",
      "met": false,
      "compare": {
        "resolved_values": {
          "ctx.payload.aggregations.bucket_results.doc_count": 0
        }
      }
    },
    "actions": []
  },
  "messages": []
}
3

Apologies had to divide my code into parts cause it wont take whole content due to restriction of letters

4

The interesting part of the response is quoted above. Your search request does not contain any hits, thus the condition is not met.

5

thanks @spinscale - if you look at my watcher then basically my anomaly score is set to 0 - and in such condition shouldn't we be expecting a confirmed hit?

{
  "trigger": {
    "schedule": {
      "interval": "119s"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          ".ml-anomalies-*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "filter": [
                {
                  "term": {
                    "job_id": "all-errors"
                  }
                },
                {
                  "range": {
                    "timestamp": {
                      "gte": "now-30m"
                    }
                  }
                },
                {
                  "terms": {
                    "result_type": [
                      "bucket",
                      "record",
                      "influencer"
                    ]
                  }
                }
              ]
            }
          },
          "aggs": {
            "bucket_results": {
              "filter": {
                "range": {
                  "anomaly_score": {
                    "gte": 0
                  }
                }
              },
              "aggs": {
                "top_bucket_hits": {
                  "top_hits": {
                    "sort": [
                      {
                        "anomaly_score": {
                          "order": "desc"
                        }
                      }
                    ],
                    "_source": {
                      "includes": [
                        "job_id",
                        "result_type",
                        "timestamp",
                        "anomaly_score",
                        "is_interim"
                      ]
                    },
                    "size": 1,
                    "script_fields": {
                      "start": {
                        "script": {
                          "lang": "painless",
                          "source": "LocalDateTime.ofEpochSecond((doc[\"timestamp\"].value.getMillis()-((doc[\"bucket_span\"].value * 1000)\n * params.padding)) / 1000, 0, ZoneOffset.UTC).toString()+\":00.000Z\"",
                          "params": {
                            "padding": 10
                          }
                        }
                      },
                      "end": {
                        "script": {
                          "lang": "painless",
                          "source": "LocalDateTime.ofEpochSecond((doc[\"timestamp\"].value.getMillis()+((doc[\"bucket_span\"].value * 1000)\n * params.padding)) / 1000, 0, ZoneOffset.UTC).toString()+\":00.000Z\"",
                          "params": {
                            "padding": 10
                          }
                        }
                      },
                      "timestamp_epoch": {
                        "script": {
                          "lang": "painless",
                          "source": """doc["timestamp"].value.getMillis()/1000"""
                        }
                      },
                      "timestamp_iso8601": {
                        "script": {
                          "lang": "painless",
                          "source": """doc["timestamp"].value"""
                        }
                      },
                      "score": {
                        "script": {
                          "lang": "painless",
                          "source": """Math.round(doc["anomaly_score"].value)"""
                        }
                      }
                    }
                  }
                }
              }
            },
            "influencer_results": {
              "filter": {
                "range": {
                  "influencer_score": {
                    "gte": 3
                  }
                }
              },
              "aggs": {
                "top_influencer_hits": {
                  "top_hits": {
                    "sort": [
                      {
                        "influencer_score": {
                          "order": "desc"
                        }
                      }
                    ],
                    "_source": {
                      "includes": [
                        "result_type",
                        "timestamp",
                        "influencer_field_name",
                        "influencer_field_value",
                        "influencer_score",
                        "isInterim"
                      ]
                    },
                    "size": 3,
                    "script_fields": {
                      "score": {
                        "script": {
                          "lang": "painless",
                          "source": """Math.round(doc["influencer_score"].value)"""
                        }
                      }
                    }
                  }
                }
              }
            },
            "record_results": {
              "filter": {
                "range": {
                  "record_score": {
                    "gte": 3
                  }
                }
              },
              "aggs": {
                "top_record_hits": {
                  "top_hits": {
                    "sort": [
                      {
                        "record_score": {
                          "order": "desc"
                        }
                      }
                    ],
                    "_source": {
                      "includes": [
                        "result_type",
                        "timestamp",
                        "record_score",
                        "is_interim",
                        "function",
                        "field_name",
                        "by_field_value",
                        "over_field_value",
                        "partition_field_value"
                      ]
                    },
                    "size": 3,
                    "script_fields": {
                      "score": {
                        "script": {
                          "lang": "painless",
                          "source": """Math.round(doc["record_score"].value)"""
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.aggregations.bucket_results.doc_count": {
        "gt": 0
      }
    }
  },
  "actions": {
    "Watcher-webhook": {
      "throttle_period_in_millis": 30000,
      "webhook": {
        "scheme": "http",
        "host": "xxxxx",
        "port": xxx,
        "method": "post",
        "path": ":/alert/{{ctx.watch_id}}",
        "params": {},
        "headers": {
          "Content-Type": "application/json"
        },
        "body": """{ "signature": "[{{ctx.execution_time}}]", "source": "xxxx","environment": "xxxx", "source_id": "xxxx [{{ctx.payload.aggregations.bucket_results.top_bucket_hits.hits.hits.0._source.job_id}}]", "ApplicationCode": "xxx", "external_id":"xxx", "agent":"xxxx", "agent_location":"xxx", "severity": "CRITICAL", "moogsoft_type": "xxx", "manager": "xxxx", "class": "application","createTicket": "1" , "description": "Alert for job [{{ctx.payload.aggregations.bucket_results.top_bucket_hits.hits.hits.0._source.job_id}}] at [{{ctx.payload.aggregations.bucket_results.top_bucket_hits.hits.hits.0.fields.timestamp_iso8601.0}}] score [{{ctx.payload.aggregations.bucket_results.top_bucket_hits.hits.hits.0.fields.score.0}}]" }"""
      }
    }
  }
}
6

If you have security enabled, than the user, who you are storing the watch as, needs to have privileges to read that index. Is that given?

You can check by trying out that query in the dev-tools console.

7

Hi @spinscale this watcher was created as a part of ML - and yes user does have access to the Index

  • I tried executing using DEV Tools and below is the output i got - knowing the watcher is already created - i simply tried to post that same watcher again
    {
    "_id" : "all-errors",
    "_version" : 38032,
    "_seq_no" : 5235590,
    "_primary_term" : 2,
    "created" : false
    }
8

Just to Add - so i created a Single Metric ML and executed watcher - it seems to work fine. This is happening with Multi Metric ML mostly - Watcher is created as a Part of ML - and at the time of "Simulation" i am making change to the parameter anomoly_score from 75 to 0 - Now this works perfectly fine if its a Single Metric watcher - How can be Simulate a Multi Metric Watcher by passing manual values? because it seems there's more than just "anomoly_score" when it comes to Multi Metric Watcher

  "aggs": {
            "bucket_results": {
              "filter": {
                "range": {
                  "anomaly_score": {
                    "gte": 0
                  }
                }
              },
9

To get a better understanding of ML results information, and how it relates to alerting may I suggest:

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.