Set alert based on Visualization

Hi,

We have created a lens visualization for showing the apdex score of APM traces and we would like to create an alert based on that score. Our knowledge in alerting and monitoring is still limited and we haven't found a solution to transform that in an alert.

I attach an image of the visualization:

And here is some extra info from the visualization:



If the score goes under 0.8 then it should trigger an alert.

Let mw know if you need any extra info. Thanks in advance.

Mario

Hi Mario, thanks for reaching out. We have this on our list of functionality to add. We are just about to release "Search threshold alerts" in Discover in 8.3 and, beyond that, this style of threshold alert based on a metric is absolutely where we want to go. Here's the issue for this improvement so you can follow along - [Lens] Alerting from Lens Visualizations · Issue #71150 · elastic/kibana · GitHub

1 Like

Hi @ghudgins

Thanks for your quick answer.
Is there a way to extract the query info of the visualization so we can set an alert while we wait for 8.3 version? When should this new version be released?

Kind regards

Mario

The query for any visualization can be seen by using the "Inspect" option in Lens. Some visualizations require multiple queries to build the final end product--you'll be able to navigate the requests used to retrieve that metric.

Here is a full list of our alerting functionality today in 8.2 - Alerting | Kibana Guide [8.2] | Elastic

The search threshold coming in 8.3 (which is due out in the next few weeks) likely does NOT satisfy your use case but it's a step in the right direction.

We definitely have it on our roadmap to address the "Alerting from Lens Visualizations" issue I linked above but don't have a concrete timeframe I can commit to here at this time.

You could use Watcher in the meantime to create this alert, per service.name

Here's a similar example (using different data). This is responsetime per airline visualization:

And here's the equivalent watch (you would need to change this to not use the simulate (_execute) endpoint and configure whatever appropriate action you want:

POST _watcher/watch/_execute
{
  "watch": {
    "trigger": {
      "schedule": {
        "interval": "5m"
      }
    },
    "input": {
      "search": {
        "request": {
          "indices": [
            "farequote"
          ],
          "body": {
            "size": 0,
            "query": {
              "bool": {
                "filter": [
                  {
                    "range": {
                      "@timestamp": {
                        "gte": "now-5m"
                      }
                    }
                  }
                ]
              }
            },
            "aggs": {
              "airlines": {
                "terms": {
                  "field": "airline",
                  "size": 20
                },
                "aggs": {
                  "max_responsetime": {
                    "max": {
                      "field": "responsetime"
                    }
                  }
                }
              }
            }
          }
        }
      }
    },
    "condition": {
      "script": """

        return ctx.payload.aggregations.airlines.buckets.stream()       
          .filter(airlines -> airlines.max_responsetime.value > 3000)               
          .count() > 0 
        """
    },
    "actions": {
      "log": {
        "transform": {
          "script": """
        return ctx.payload.aggregations.airlines.buckets.stream()       
          .filter(airlines -> airlines.max_responsetime.value > 3000)               
          .collect(Collectors.toList());
          """
        },
        "logging": {
          "text": """
		{{#ctx.payload._value}}
		airline={{key}} exceeded 3sec threshold with responsetime={{max_responsetime.value}}
		{{/ctx.payload._value}}
		"""
        }
      }
    }
  }
}

Sample result would look like:

		airline=SWR exceeded 3sec threshold with responsetime=4038.84
		airline=NKS exceeded 3sec threshold with responsetime=10334.94

Thanks @richcollier!

We will try to use watcher in the meantime. If we succeed we will let you know :wink:

Best regards

Mario

Hi @richcollier

I made some changes to your watcher example but I think something seems to be failing.

This is my _watcher post

POST _watcher/watch/_execute
{
  "watch": {
    "trigger": {
      "schedule": {
        "interval": "5m"
      }
    },
    "input": {
      "search": {
        "request": {
          "indices": [
            "traces-apm*"
          ],
          "body": {
            "size": 0,
            "query": {
              "bool": {
                "filter": [
                  {
                    "range": {
                      "@timestamp": {
                        "gte": "now-5m"
                      }
                    }
                  }
                ]
              }
            },
            "aggs": {
              "services": {
                "terms": {
                  "field": "service.name",
                  "size": 20
                },
                "aggs": {
                  "avg_apdexscore": {
                    "avg": {
                      "field": "labels.apdexscore"
                    }
                  }
                }
              }
            }
          }
        }
      }
    },
    "condition": {
      "script": """

        return ctx.payload.aggregations.services.buckets.stream()       
          .filter(services -> services.avg_apdexscore.value < 0.8)               
          .count() > 0 
        """
    },
    "actions": {
      "log": {
        "transform": {
          "script": """
        return ctx.payload.aggregations.services.buckets.stream()       
          .filter(services -> services.avg_apdexscore.value > 3000)               
          .collect(Collectors.toList());
          """
        },
        "logging": {
          "text": """
		{{#ctx.payload._value}}
		service.name={{key}} exceeded 3sec threshold with labels.apdexscore={{avg_apdexscore.value}}
		{{/ctx.payload._value}}
		"""
        }
      }
    }
  }
}

And this is the result

{
  "_id" : "_inlined__**************",
  "watch_record" : {
    "watch_id" : "_inlined_",
    "node" : "**************",
    "state" : "failed",
    "@timestamp" : "2022-06-21T10:44:26.439018514Z",
    "user" : "**************",
    "status" : {
      "state" : {
        "active" : true,
        "timestamp" : "2022-06-21T10:44:26.438Z"
      },
      "actions" : {
        "log" : {
          "ack" : {
            "timestamp" : "2022-06-21T10:44:26.438Z",
            "state" : "awaits_successful_execution"
          }
        }
      },
      "execution_state" : "failed",
      "version" : -1
    },
    "trigger_event" : {
      "type" : "manual",
      "triggered_time" : "2022-06-21T10:44:26.439Z",
      "manual" : {
        "schedule" : {
          "scheduled_time" : "2022-06-21T10:44:26.439Z"
        }
      }
    },
    "input" : {
      "search" : {
        "request" : {
          "search_type" : "query_then_fetch",
          "indices" : [
            "traces-apm*"
          ],
          "rest_total_hits_as_int" : true,
          "body" : {
            "size" : 0,
            "query" : {
              "bool" : {
                "filter" : [
                  {
                    "range" : {
                      "@timestamp" : {
                        "gte" : "now-5m"
                      }
                    }
                  }
                ]
              }
            },
            "aggs" : {
              "services" : {
                "terms" : {
                  "field" : "service.name",
                  "size" : 20
                },
                "aggs" : {
                  "avg_apdexscore" : {
                    "avg" : {
                      "field" : "labels.apdexscore"
                    }
                  }
                }
              }
            }
          }
        }
      }
    },
    "condition" : {
      "script" : {
        "source" : """

        return ctx.payload.aggregations.services.buckets.stream()       
          .filter(services -> services.avg_apdexscore.value < 0.8)               
          .count() > 0 
        """,
        "lang" : "painless"
      }
    },
    "result" : {
      "execution_time" : "2022-06-21T10:44:26.439Z",
      "execution_duration" : 3,
      "input" : {
        "type" : "search",
        "status" : "success",
        "payload" : {
          "_shards" : {
            "total" : 12,
            "failed" : 0,
            "successful" : 12,
            "skipped" : 0
          },
          "hits" : {
            "hits" : [ ],
            "total" : 60,
            "max_score" : null
          },
          "took" : 2,
          "timed_out" : false,
          "aggregations" : {
            "services" : {
              "doc_count_error_upper_bound" : 0,
              "sum_other_doc_count" : 0,
              "buckets" : [
                {
                  "doc_count" : 60,
                  "avg_apdexscore" : {
                    "value" : null
                  },
                  "key" : "staging-jucy"
                }
              ]
            }
          }
        },
        "search" : {
          "request" : {
            "search_type" : "query_then_fetch",
            "indices" : [
              "traces-apm*"
            ],
            "rest_total_hits_as_int" : true,
            "body" : {
              "size" : 0,
              "query" : {
                "bool" : {
                  "filter" : [
                    {
                      "range" : {
                        "@timestamp" : {
                          "gte" : "now-5m"
                        }
                      }
                    }
                  ]
                }
              },
              "aggs" : {
                "services" : {
                  "terms" : {
                    "field" : "service.name",
                    "size" : 20
                  },
                  "aggs" : {
                    "avg_apdexscore" : {
                      "avg" : {
                        "field" : "labels.apdexscore"
                      }
                    }
                  }
                }
              }
            }
          }
        }
      },
      "actions" : [ ]
    },
    "exception" : {
      "type" : "script_exception",
      "reason" : "runtime error",
      "script_stack" : [
        """services -> services.avg_apdexscore.value < 0.8)               
          .count() > 0 
        """,
        "                                   ^---- HERE"
      ],
      "script" : " ...",
      "lang" : "painless",
      "position" : {
        "offset" : 128,
        "start" : 93,
        "end" : 189
      },
      "caused_by" : {
        "type" : "null_pointer_exception",
        "reason" : """Cannot invoke "Object.getClass()" because "leftObject" is null""",
        "stack_trace" : """java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "leftObject" is null
	at org.elasticsearch.painless.DefBootstrap$MIC.checkLHS(DefBootstrap.java:416)
	at org.elasticsearch.painless.PainlessScript$Script.lambda$synthetic$0( ...:129)
	at org.elasticsearch.painless.PainlessScript$Script$$Lambda0.test(Unknown Source)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:178)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:258)
	at java.base/java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:248)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.count(ReferencePipeline.java:709)
	at org.elasticsearch.painless.PainlessScript$Script.execute( ...:168)
	at org.elasticsearch.xpack.watcher.condition.ScriptCondition.doExecute(ScriptCondition.java:65)
	at org.elasticsearch.xpack.watcher.condition.ScriptCondition.execute(ScriptCondition.java:60)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:542)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:342)
	at org.elasticsearch.xpack.watcher.transport.actions.TransportExecuteWatchAction$1.doRun(TransportExecuteWatchAction.java:189)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:669)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
"""
      },
      "stack_trace" : """org.elasticsearch.script.ScriptException: runtime error
	at org.elasticsearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:85)
	at org.elasticsearch.painless.PainlessScript$Script.execute( ...:1)
	at org.elasticsearch.xpack.watcher.condition.ScriptCondition.doExecute(ScriptCondition.java:65)
	at org.elasticsearch.xpack.watcher.condition.ScriptCondition.execute(ScriptCondition.java:60)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:542)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:342)
	at org.elasticsearch.xpack.watcher.transport.actions.TransportExecuteWatchAction$1.doRun(TransportExecuteWatchAction.java:189)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:669)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "leftObject" is null
	at org.elasticsearch.painless.DefBootstrap$MIC.checkLHS(DefBootstrap.java:416)
	at org.elasticsearch.painless.PainlessScript$Script.lambda$synthetic$0( ...:129)
	at org.elasticsearch.painless.PainlessScript$Script$$Lambda0.test(Unknown Source)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:178)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:258)
	at java.base/java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:248)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.count(ReferencePipeline.java:709)
	at org.elasticsearch.painless.PainlessScript$Script.execute( ...:168)
	... 11 more
"""
    }
  }
}

Not sure if there is something missing or I ddi something wrong

Kind Regards

Mario

Let's take Watcher out of the picture for a minute. What does the output look like for the query:

GET traces-apm*/_search
{
  "size": 0,
  "aggs": {
    "services": {
      "terms": {
        "field": "service.name",
        "size": 20
      },
      "aggs": {
        "avg_apdexscore": {
          "avg": {
            "field": "labels.apdexscore"
          }
        }
      }
    }
  }
}

I did run it in the wrong deployment :man_facepalming: Sorry about that!!

I did it now in the right place and it looks much better but still with some errors. The last query you sent works :slightly_smiling_face:

This is the result of the watcher post:

{
  "_id" : "_inlined__*************",
  "watch_record" : {
    "watch_id" : "_inlined_",
    "node" : "*************",
    "state" : "failed",
    "@timestamp" : "2022-06-21T14:56:49.136247915Z",
    "user" : "*************",
    "status" : {
      "state" : {
        "active" : true,
        "timestamp" : "2022-06-21T14:56:49.128Z"
      },
      "actions" : {
        "log" : {
          "ack" : {
            "timestamp" : "2022-06-21T14:56:49.128Z",
            "state" : "awaits_successful_execution"
          }
        }
      },
      "execution_state" : "failed",
      "version" : -1
    },
    "trigger_event" : {
      "type" : "manual",
      "triggered_time" : "2022-06-21T14:56:49.136Z",
      "manual" : {
        "schedule" : {
          "scheduled_time" : "2022-06-21T14:56:49.136Z"
        }
      }
    },
    "input" : {
      "search" : {
        "request" : {
          "search_type" : "query_then_fetch",
          "indices" : [
            "traces-apm*"
          ],
          "rest_total_hits_as_int" : true,
          "body" : {
            "size" : 0,
            "query" : {
              "bool" : {
                "filter" : [
                  {
                    "range" : {
                      "@timestamp" : {
                        "gte" : "now-5m"
                      }
                    }
                  }
                ]
              }
            },
            "aggs" : {
              "services" : {
                "terms" : {
                  "field" : "service.name",
                  "size" : 20
                },
                "aggs" : {
                  "avg_apdexscore" : {
                    "avg" : {
                      "field" : "labels.apdexscore"
                    }
                  }
                }
              }
            }
          }
        }
      }
    },
    "condition" : {
      "script" : {
        "source" : """

        return ctx.payload.aggregations.services.buckets.stream()       
          .filter(services -> services.avg_apdexscore.value < 0.8)               
          .count() > 0 
        """,
        "lang" : "painless"
      }
    },
    "result" : {
      "execution_time" : "2022-06-21T14:56:49.136Z",
      "execution_duration" : 1279,
      "input" : {
        "type" : "search",
        "status" : "success",
        "payload" : {
          "_shards" : {
            "total" : 51,
            "failed" : 0,
            "successful" : 51,
            "skipped" : 0
          },
          "hits" : {
            "hits" : [ ],
            "total" : 10000,
            "max_score" : null
          },
          "took" : 1269,
          "timed_out" : false,
          "aggregations" : {
            "services" : {
              "doc_count_error_upper_bound" : 0,
              "sum_other_doc_count" : 6420,
              "buckets" : [
                {
                  "doc_count" : 120052,
                  "avg_apdexscore" : {
                    "value" : 0.9906687402799378
                  },
                  "key" : "service1"
                },
                {
                  "doc_count" : 61031,
                  "avg_apdexscore" : {
                    "value" : null
                  },
                  "key" : "service2"
                },
                {
                  "doc_count" : 36053,
                  "avg_apdexscore" : {
                    "value" : 0.9924242424242424
                  },
                  "key" : "service3"
                },
                {
                  "doc_count" : 33921,
                  "avg_apdexscore" : {
                    "value" : 0.9965166908563134
                  },
                  "key" : "service4"
                },
                {
                  "doc_count" : 28276,
                  "avg_apdexscore" : {
                    "value" : 0.9984508133230054
                  },
                  "key" : "service5"
                },
                {
                  "doc_count" : 22541,
                  "avg_apdexscore" : {
                    "value" : 0.9969179706021811
                  },
                  "key" : "service6"
                },
                {
                  "doc_count" : 20524,
                  "avg_apdexscore" : {
                    "value" : 0.9965139442231076
                  },
                  "key" : "service7"
                },
                {
                  "doc_count" : 17962,
                  "avg_apdexscore" : {
                    "value" : null
                  },
                  "key" : "service8"
                },
                {
                  "doc_count" : 14103,
                  "avg_apdexscore" : {
                    "value" : null
                  },
                  "key" : "service9"
                },
                {
                  "doc_count" : 13839,
                  "avg_apdexscore" : {
                    "value" : 0.9953401677539608
                  },
                  "key" : "service10"
                },
                {
                  "doc_count" : 13617,
                  "avg_apdexscore" : {
                    "value" : 0.9990341274951706
                  },
                  "key" : "service11"
                },
                {
                  "doc_count" : 12318,
                  "avg_apdexscore" : {
                    "value" : null
                  },
                  "key" : "service12"
                },
                {
                  "doc_count" : 12066,
                  "avg_apdexscore" : {
                    "value" : 0.9987725040916531
                  },
                  "key" : "service13"
                },
                {
                  "doc_count" : 10642,
                  "avg_apdexscore" : {
                    "value" : null
                  },
                  "key" : "service14"
                },
                {
                  "doc_count" : 9688,
                  "avg_apdexscore" : {
                    "value" : 0.9993946731234867
                  },
                  "key" : "service15"
                },
                {
                  "doc_count" : 8857,
                  "avg_apdexscore" : {
                    "value" : null
                  },
                  "key" : "service16"
                },
                {
                  "doc_count" : 5326,
                  "avg_apdexscore" : {
                    "value" : null
                  },
                  "key" : "service17"
                },
                {
                  "doc_count" : 5322,
                  "avg_apdexscore" : {
                    "value" : null
                  },
                  "key" : "service18"
                },
                {
                  "doc_count" : 5156,
                  "avg_apdexscore" : {
                    "value" : null
                  },
                  "key" : "service19"
                },
                {
                  "doc_count" : 3863,
                  "avg_apdexscore" : {
                    "value" : null
                  },
                  "key" : "service20"
                }
              ]
            }
          }
        },
        "search" : {
          "request" : {
            "search_type" : "query_then_fetch",
            "indices" : [
              "traces-apm*"
            ],
            "rest_total_hits_as_int" : true,
            "body" : {
              "size" : 0,
              "query" : {
                "bool" : {
                  "filter" : [
                    {
                      "range" : {
                        "@timestamp" : {
                          "gte" : "now-5m"
                        }
                      }
                    }
                  ]
                }
              },
              "aggs" : {
                "services" : {
                  "terms" : {
                    "field" : "service.name",
                    "size" : 20
                  },
                  "aggs" : {
                    "avg_apdexscore" : {
                      "avg" : {
                        "field" : "labels.apdexscore"
                      }
                    }
                  }
                }
              }
            }
          }
        }
      },
      "actions" : [ ]
    },
    "exception" : {
      "type" : "script_exception",
      "reason" : "runtime error",
      "script_stack" : [
        """services -> services.avg_apdexscore.value < 0.8)               
          .count() > 0 
        """,
        "                                   ^---- HERE"
      ],
      "script" : " ...",
      "lang" : "painless",
      "position" : {
        "offset" : 128,
        "start" : 93,
        "end" : 189
      },
      "caused_by" : {
        "type" : "null_pointer_exception",
        "reason" : """Cannot invoke "Object.getClass()" because "leftObject" is null""",
        "stack_trace" : """java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "leftObject" is null
	at org.elasticsearch.painless.DefBootstrap$MIC.checkLHS(DefBootstrap.java:416)
	at org.elasticsearch.painless.PainlessScript$Script.lambda$synthetic$0( ...:129)
	at org.elasticsearch.painless.PainlessScript$Script$$Lambda0.test(Unknown Source)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:178)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:258)
	at java.base/java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:248)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.count(ReferencePipeline.java:709)
	at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:732)
	at org.elasticsearch.painless.DefBootstrap$PIC.fallback(DefBootstrap.java:239)
	at org.elasticsearch.painless.PainlessScript$Script.execute( ...:168)
	at org.elasticsearch.xpack.watcher.condition.ScriptCondition.doExecute(ScriptCondition.java:65)
	at org.elasticsearch.xpack.watcher.condition.ScriptCondition.execute(ScriptCondition.java:60)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:542)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:342)
	at org.elasticsearch.xpack.watcher.transport.actions.TransportExecuteWatchAction$1.doRun(TransportExecuteWatchAction.java:189)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:669)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
"""
      },
      "stack_trace" : """org.elasticsearch.script.ScriptException: runtime error
	at org.elasticsearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:85)
	at org.elasticsearch.painless.PainlessScript$Script.execute( ...:1)
	at org.elasticsearch.xpack.watcher.condition.ScriptCondition.doExecute(ScriptCondition.java:65)
	at org.elasticsearch.xpack.watcher.condition.ScriptCondition.execute(ScriptCondition.java:60)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:542)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:342)
	at org.elasticsearch.xpack.watcher.transport.actions.TransportExecuteWatchAction$1.doRun(TransportExecuteWatchAction.java:189)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:669)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "leftObject" is null
	at org.elasticsearch.painless.DefBootstrap$MIC.checkLHS(DefBootstrap.java:416)
	at org.elasticsearch.painless.PainlessScript$Script.lambda$synthetic$0( ...:129)
	at org.elasticsearch.painless.PainlessScript$Script$$Lambda0.test(Unknown Source)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:178)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:258)
	at java.base/java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:248)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.count(ReferencePipeline.java:709)
	at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:732)
	at org.elasticsearch.painless.DefBootstrap$PIC.fallback(DefBootstrap.java:239)
	at org.elasticsearch.painless.PainlessScript$Script.execute( ...:168)
	... 11 more
"""
    }
  }
}

Thanks

In case you also need the query result here it is:

{
  "took" : 8327,
  "timed_out" : false,
  "_shards" : {
    "total" : 50,
    "successful" : 50,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "services" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 37060702,
      "buckets" : [
        {
          "key" : "service1",
          "doc_count" : 751728057,
          "avg_apdexscore" : {
            "value" : 0.9900452781045794
          }
        },
        {
          "key" : "service2",
          "doc_count" : 368399400,
          "avg_apdexscore" : {
            "value" : null
          }
        },
        {
          "key" : "service3",
          "doc_count" : 187379449,
          "avg_apdexscore" : {
            "value" : 0.9970768098569803
          }
        },
        {
          "key" : "service4",
          "doc_count" : 166119692,
          "avg_apdexscore" : {
            "value" : 0.9981641091946719
          }
        },
        {
          "key" : "service5",
          "doc_count" : 140902539,
          "avg_apdexscore" : {
            "value" : 0.9905411018902832
          }
        },
        {
          "key" : "service6",
          "doc_count" : 131635318,
          "avg_apdexscore" : {
            "value" : 0.9977093144852212
          }
        },
        {
          "key" : "service7",
          "doc_count" : 113643554,
          "avg_apdexscore" : {
            "value" : 0.9985224096879122
          }
        },
        {
          "key" : "service8",
          "doc_count" : 98480453,
          "avg_apdexscore" : {
            "value" : 0.998956908884416
          }
        },
        {
          "key" : "service9",
          "doc_count" : 97515652,
          "avg_apdexscore" : {
            "value" : null
          }
        },
        {
          "key" : "service10",
          "doc_count" : 84340547,
          "avg_apdexscore" : {
            "value" : 0.9955444573071522
          }
        },
        {
          "key" : "service11",
          "doc_count" : 74934287,
          "avg_apdexscore" : {
            "value" : null
          }
        },
        {
          "key" : "service12",
          "doc_count" : 72040453,
          "avg_apdexscore" : {
            "value" : 0.99558658505802
          }
        },
        {
          "key" : "service13",
          "doc_count" : 69954389,
          "avg_apdexscore" : {
            "value" : 0.9972449236072635
          }
        },
        {
          "key" : "service14",
          "doc_count" : 63052293,
          "avg_apdexscore" : {
            "value" : null
          }
        },
        {
          "key" : "service15",
          "doc_count" : 54924435,
          "avg_apdexscore" : {
            "value" : null
          }
        },
        {
          "key" : "service16",
          "doc_count" : 51429637,
          "avg_apdexscore" : {
            "value" : null
          }
        },
        {
          "key" : "service17",
          "doc_count" : 41927901,
          "avg_apdexscore" : {
            "value" : null
          }
        },
        {
          "key" : "service18",
          "doc_count" : 41654399,
          "avg_apdexscore" : {
            "value" : null
          }
        },
        {
          "key" : "service19",
          "doc_count" : 32813611,
          "avg_apdexscore" : {
            "value" : null
          }
        },
        {
          "key" : "service20",
          "doc_count" : 29704495,
          "avg_apdexscore" : {
            "value" : 0.9965565582521134
          }
        }
      ]
    }
  }
}

Your problem is that some of your services have null data:

If this is expected, we'll just need to modify the code to protect against the null comparison...

Hi @richcollier

I was able to fix the null values. There was a missing processor script for one of the ingest pipelines of the apm indices.

{
  "_id" : "_inlined__**************",
  "watch_record" : {
    "watch_id" : "_inlined_",
    "node" : "**************",
    "state" : "executed",
    "@timestamp" : "2022-06-22T07:36:53.566141447Z",
    "user" : "**************",
    "status" : {
      "state" : {
        "active" : true,
        "timestamp" : "2022-06-22T07:36:53.565Z"
      },
      "last_checked" : "2022-06-22T07:36:53.566Z",
      "last_met_condition" : "2022-06-22T07:36:53.566Z",
      "actions" : {
        "log" : {
          "ack" : {
            "timestamp" : "2022-06-22T07:36:53.566Z",
            "state" : "ackable"
          },
          "last_execution" : {
            "timestamp" : "2022-06-22T07:36:53.566Z",
            "successful" : true
          },
          "last_successful_execution" : {
            "timestamp" : "2022-06-22T07:36:53.566Z",
            "successful" : true
          }
        }
      },
      "execution_state" : "executed",
      "version" : -1
    },
    "trigger_event" : {
      "type" : "manual",
      "triggered_time" : "2022-06-22T07:36:53.566Z",
      "manual" : {
        "schedule" : {
          "scheduled_time" : "2022-06-22T07:36:53.566Z"
        }
      }
    },
    "input" : {
      "search" : {
        "request" : {
          "search_type" : "query_then_fetch",
          "indices" : [
            "traces-apm*"
          ],
          "rest_total_hits_as_int" : true,
          "body" : {
            "size" : 0,
            "query" : {
              "bool" : {
                "filter" : [
                  {
                    "range" : {
                      "@timestamp" : {
                        "gte" : "now-5m"
                      }
                    }
                  }
                ]
              }
            },
            "aggs" : {
              "services" : {
                "terms" : {
                  "field" : "service.name",
                  "size" : 30
                },
                "aggs" : {
                  "avg_apdexscore" : {
                    "avg" : {
                      "field" : "labels.apdexscore"
                    }
                  }
                }
              }
            }
          }
        }
      }
    },
    "condition" : {
      "script" : {
        "source" : """

        return ctx.payload.aggregations.services.buckets.stream()       
          .filter(services -> services.avg_apdexscore.value < 0.8)               
          .count() > 0 
        """,
        "lang" : "painless"
      }
    },
    "result" : {
      "execution_time" : "2022-06-22T07:36:53.566Z",
      "execution_duration" : 68,
      "input" : {
        "type" : "search",
        "status" : "success",
        "payload" : {
          "_shards" : {
            "total" : 50,
            "failed" : 0,
            "successful" : 50,
            "skipped" : 0
          },
          "hits" : {
            "hits" : [ ],
            "total" : 10000,
            "max_score" : null
          },
          "took" : 66,
          "timed_out" : false,
          "aggregations" : {
            "services" : {
              "doc_count_error_upper_bound" : 0,
              "sum_other_doc_count" : 0,
              "buckets" : [
                {
                  "doc_count" : 99528,
                  "avg_apdexscore" : {
                    "value" : 0.991608046402567
                  },
                  "key" : "service1"
                },
                {
                  "doc_count" : 52511,
                  "avg_apdexscore" : {
                    "value" : 0.8421935102081073
                  },
                  "key" : "service2"
                },
                {
                  "doc_count" : 28721,
                  "avg_apdexscore" : {
                    "value" : 0.9943089430894309
                  },
                  "key" : "service3"
                },
                {
                  "doc_count" : 25580,
                  "avg_apdexscore" : {
                    "value" : 0.9958333333333333
                  },
                  "key" : "service4"
                },
                {
                  "doc_count" : 25494,
                  "avg_apdexscore" : {
                    "value" : 0.9980323567993004
                  },
                  "key" : "service5"
                },
                {
                  "doc_count" : 15721,
                  "avg_apdexscore" : {
                    "value" : 1.0
                  },
                  "key" : "service6"
                },
                {
                  "doc_count" : 14718,
                  "avg_apdexscore" : {
                    "value" : 0.9984394506866417
                  },
                  "key" : "service7"
                },
                {
                  "doc_count" : 14430,
                  "avg_apdexscore" : {
                    "value" : 0.7919758812615956
                  },
                  "key" : "service8"
                },
                {
                  "doc_count" : 14377,
                  "avg_apdexscore" : {
                    "value" : 0.9991749174917491
                  },
                  "key" : "service9"
                },
                {
                  "doc_count" : 14250,
                  "avg_apdexscore" : {
                    "value" : 0.9990221642764016
                  },
                  "key" : "service10"
                },
                {
                  "doc_count" : 12484,
                  "avg_apdexscore" : {
                    "value" : 0.812205466540999
                  },
                  "key" : "service11"
                },
                {
                  "doc_count" : 10796,
                  "avg_apdexscore" : {
                    "value" : 0.8555351401010565
                  },
                  "key" : "service12"
                },
                {
                  "doc_count" : 9479,
                  "avg_apdexscore" : {
                    "value" : 0.9973375931842385
                  },
                  "key" : "service13"
                },
                {
                  "doc_count" : 8413,
                  "avg_apdexscore" : {
                    "value" : 0.9965277777777778
                  },
                  "key" : "service14"
                },
                {
                  "doc_count" : 7784,
                  "avg_apdexscore" : {
                    "value" : 0.8818503241013553
                  },
                  "key" : "service15"
                },
                {
                  "doc_count" : 7688,
                  "avg_apdexscore" : {
                    "value" : 0.8477975016436555
                  },
                  "key" : "service16"
                },
                {
                  "doc_count" : 6477,
                  "avg_apdexscore" : {
                    "value" : 0.8296387520525451
                  },
                  "key" : "service17"
                },
                {
                  "doc_count" : 5683,
                  "avg_apdexscore" : {
                    "value" : 0.8875106928999145
                  },
                  "key" : "service18"
                },
                {
                  "doc_count" : 4275,
                  "avg_apdexscore" : {
                    "value" : 0.9988317757009346
                  },
                  "key" : "service19"
                },
                {
                  "doc_count" : 3455,
                  "avg_apdexscore" : {
                    "value" : 0.8892005610098177
                  },
                  "key" : "service20"
                },
                {
                  "doc_count" : 2700,
                  "avg_apdexscore" : {
                    "value" : 0.8570205479452054
                  },
                  "key" : "service21"
                },
                {
                  "doc_count" : 2600,
                  "avg_apdexscore" : {
                    "value" : 0.8529411764705882
                  },
                  "key" : "service22"
                },
                {
                  "doc_count" : 278,
                  "avg_apdexscore" : {
                    "value" : 0.996
                  },
                  "key" : "service23"
                },
                {
                  "doc_count" : 30,
                  "avg_apdexscore" : {
                    "value" : 1.0
                  },
                  "key" : "service24"
                }
              ]
            }
          }
        },
        "search" : {
          "request" : {
            "search_type" : "query_then_fetch",
            "indices" : [
              "traces-apm*"
            ],
            "rest_total_hits_as_int" : true,
            "body" : {
              "size" : 0,
              "query" : {
                "bool" : {
                  "filter" : [
                    {
                      "range" : {
                        "@timestamp" : {
                          "gte" : "now-5m"
                        }
                      }
                    }
                  ]
                }
              },
              "aggs" : {
                "services" : {
                  "terms" : {
                    "field" : "service.name",
                    "size" : 30
                  },
                  "aggs" : {
                    "avg_apdexscore" : {
                      "avg" : {
                        "field" : "labels.apdexscore"
                      }
                    }
                  }
                }
              }
            }
          }
        }
      },
      "condition" : {
        "type" : "script",
        "status" : "success",
        "met" : true
      },
      "actions" : [
        {
          "id" : "log",
          "type" : "logging",
          "status" : "success",
          "transform" : {
            "type" : "script",
            "status" : "success",
            "payload" : {
              "_value" : [ ]
            }
          },
          "logging" : {
            "logged_text" : """
		"""
          }
        }
      ]
    },
    "messages" : [ ]
  }
}

To summarise, this calculates the average apdex score by service during the last 5 minutes, right?

Kind regards

Yes it does. I see that you're not getting any output in the logged_text and then I noticed that you probably have a typo in your action:

It should not be > 3000 (that was my example). It should be < 0.8

Hi @richcollier

I was able to create the watcher and simulate some notifications. I modified the actions to notify to slack when there is a value below 0.9 (It will be 0.8 once it works as expected).

Here is the JSON content of the watcher

{
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "traces-apm*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "filter": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-5m"
                    }
                  }
                }
              ]
            }
          },
          "aggs": {
            "services": {
              "terms": {
                "field": "service.name",
                "size": 30
              },
              "aggs": {
                "avg_apdexscore": {
                  "avg": {
                    "field": "labels.apdexscore"
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "script": {
      "source": """
      return ctx.payload.aggregations.services.buckets.stream()       
        .filter(services -> services.avg_apdexscore.value < 0.9)               
        .count() > 0 
      """,
      "lang": "painless"
    }
  },
  "actions": {
    "notify-slack": {
      "throttle_period_in_millis": 300000,
      "transform": {
        "script": {
          "source": """
        return ctx.payload.aggregations.services.buckets.stream()       
        .filter(services -> services.avg_apdexscore.value < 0.9)               
        .collect(Collectors.toList());
        """,
          "lang": "painless"
        }
      },
      "slack": {
        "message": {
          "text": """{{#ctx.payload._value}} service.name={{key}} is below 0.9 with labels.apdexscore={{avg_apdexscore.value}} 
          {{/ctx.payload._value}}"""
        }
      }
    }
  }
}

And this is a Simulate watch

{
  "watch_id": "_inlined_",
  "node": "**************",
  "state": "executed",
  "@timestamp": "2022-06-22T13:30:32.298895082Z",
  "user": "**************",
  "status": {
    "state": {
      "active": true,
      "timestamp": "2022-06-22T13:30:32.298Z"
    },
    "last_checked": "2022-06-22T13:30:32.298Z",
    "last_met_condition": "2022-06-22T13:30:32.298Z",
    "actions": {
      "notify-slack": {
        "ack": {
          "timestamp": "2022-06-22T13:30:32.298Z",
          "state": "ackable"
        },
        "last_execution": {
          "timestamp": "2022-06-22T13:30:32.298Z",
          "successful": true
        },
        "last_successful_execution": {
          "timestamp": "2022-06-22T13:30:32.298Z",
          "successful": true
        }
      }
    },
    "execution_state": "executed",
    "version": -1
  },
  "trigger_event": {
    "type": "manual",
    "triggered_time": "2022-06-22T13:30:32.298Z",
    "manual": {
      "schedule": {
        "scheduled_time": "2022-06-22T13:30:32.298Z"
      }
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "traces-apm*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "filter": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-5m"
                    }
                  }
                }
              ]
            }
          },
          "aggs": {
            "services": {
              "terms": {
                "field": "service.name",
                "size": 30
              },
              "aggs": {
                "avg_apdexscore": {
                  "avg": {
                    "field": "labels.apdexscore"
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "script": {
      "source": "\n      return ctx.payload.aggregations.services.buckets.stream()       \n        .filter(services -> services.avg_apdexscore.value < 0.9)               \n        .count() > 0 \n      ",
      "lang": "painless"
    }
  },
  "metadata": {
    "name": "Apdex score by service",
    "xpack": {
      "type": "json"
    }
  },
  "result": {
    "execution_time": "2022-06-22T13:30:32.298Z",
    "execution_duration": 2952,
    "input": {
      "type": "search",
      "status": "success",
      "payload": {
        "_shards": {
          "total": 52,
          "failed": 0,
          "successful": 52,
          "skipped": 0
        },
        "hits": {
          "hits": [],
          "total": 10000,
          "max_score": null
        },
        "took": 2950,
        "timed_out": false,
        "aggregations": {
          "services": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "doc_count": 124695,
                "avg_apdexscore": {
                  "value": 0.9914766892585747
                },
                "key": "service1"
              },
              {
                "doc_count": 65515,
                "avg_apdexscore": {
                  "value": 0.7985335195530726
                },
                "key": "service2"
              },
              {
                "doc_count": 38725,
                "avg_apdexscore": {
                  "value": 0.9988324576765908
                },
                "key": "service3"
              },
              {
                "doc_count": 34317,
                "avg_apdexscore": {
                  "value": 0.9955449955449955
                },
                "key": "service4"
              },
              {
                "doc_count": 22326,
                "avg_apdexscore": {
                  "value": 0.9893672199170125
                },
                "key": "service5"
              },
              {
                "doc_count": 19625,
                "avg_apdexscore": {
                  "value": 0.9994541484716157
                },
                "key": "service6"
              },
              {
                "doc_count": 18998,
                "avg_apdexscore": {
                  "value": 0.9980136208853575
                },
                "key": "service7"
              },
              {
                "doc_count": 18692,
                "avg_apdexscore": {
                  "value": 0.8258392362180474
                },
                "key": "service8"
              },
              {
                "doc_count": 17580,
                "avg_apdexscore": {
                  "value": 0.7917981072555205
                },
                "key": "service9"
              },
              {
                "doc_count": 12413,
                "avg_apdexscore": {
                  "value": 0.9996461429582448
                },
                "key": "service10"
              },
              {
                "doc_count": 10132,
                "avg_apdexscore": {
                  "value": 0.9968916518650088
                },
                "key": "service11"
              },
              {
                "doc_count": 9431,
                "avg_apdexscore": {
                  "value": 0.9986376021798365
                },
                "key": "service12"
              },
              {
                "doc_count": 9045,
                "avg_apdexscore": {
                  "value": 0.8404023926046764
                },
                "key": "service13"
              },
              {
                "doc_count": 8522,
                "avg_apdexscore": {
                  "value": 0.9949109414758269
                },
                "key": "service14"
              },
              {
                "doc_count": 8322,
                "avg_apdexscore": {
                  "value": 0.8571428571428571
                },
                "key": "service15"
              },
              {
                "doc_count": 7167,
                "avg_apdexscore": {
                  "value": 0.8769911504424779
                },
                "key": "service16"
              },
              {
                "doc_count": 5987,
                "avg_apdexscore": {
                  "value": 0.8183503243744208
                },
                "key": "service17"
              },
              {
                "doc_count": 4937,
                "avg_apdexscore": {
                  "value": 0.8734817813765182
                },
                "key": "service18"
              },
              {
                "doc_count": 4350,
                "avg_apdexscore": {
                  "value": 0.867335562987737
                },
                "key": "service19"
              },
              {
                "doc_count": 4080,
                "avg_apdexscore": {
                  "value": 0.9984984984984985
                },
                "key": "service20"
              },
              {
                "doc_count": 3179,
                "avg_apdexscore": {
                  "value": 0.8579545454545454
                },
                "key": "service21"
              },
              {
                "doc_count": 2190,
                "avg_apdexscore": {
                  "value": 0.8739224137931034
                },
                "key": "service22"
              },
              {
                "doc_count": 780,
                "avg_apdexscore": {
                  "value": 0.991701244813278
                },
                "key": "service23"
              },
              {
                "doc_count": 30,
                "avg_apdexscore": {
                  "value": 1
                },
                "key": "service24"
              }
            ]
          }
        }
      },
      "search": {
        "request": {
          "search_type": "query_then_fetch",
          "indices": [
            "traces-apm*"
          ],
          "rest_total_hits_as_int": true,
          "body": {
            "size": 0,
            "query": {
              "bool": {
                "filter": [
                  {
                    "range": {
                      "@timestamp": {
                        "gte": "now-5m"
                      }
                    }
                  }
                ]
              }
            },
            "aggs": {
              "services": {
                "terms": {
                  "field": "service.name",
                  "size": 30
                },
                "aggs": {
                  "avg_apdexscore": {
                    "avg": {
                      "field": "labels.apdexscore"
                    }
                  }
                }
              }
            }
          }
        }
      }
    },
    "condition": {
      "type": "script",
      "status": "success",
      "met": true
    },
    "actions": [
      {
        "id": "notify-slack",
        "type": "slack",
        "status": "simulated",
        "transform": {
          "type": "script",
          "status": "success",
          "payload": {
            "_value": [
              {
                "doc_count": 65515,
                "avg_apdexscore": {
                  "value": 0.7985335195530726
                },
                "key": "service2"
              },
              {
                "doc_count": 18692,
                "avg_apdexscore": {
                  "value": 0.8258392362180474
                },
                "key": "service3"
              },
              {
                "doc_count": 17580,
                "avg_apdexscore": {
                  "value": 0.7917981072555205
                },
                "key": "service4"
              },
              {
                "doc_count": 9045,
                "avg_apdexscore": {
                  "value": 0.8404023926046764
                },
                "key": "service7"
              },
              {
                "doc_count": 8322,
                "avg_apdexscore": {
                  "value": 0.8571428571428571
                },
                "key": "service9"
              },
              {
                "doc_count": 7167,
                "avg_apdexscore": {
                  "value": 0.8769911504424779
                },
                "key": "service10"
              },
              {
                "doc_count": 5987,
                "avg_apdexscore": {
                  "value": 0.8183503243744208
                },
                "key": "service16"
              },
              {
                "doc_count": 4937,
                "avg_apdexscore": {
                  "value": 0.8734817813765182
                },
                "key": "service17"
              },
              {
                "doc_count": 4350,
                "avg_apdexscore": {
                  "value": 0.867335562987737
                },
                "key": "service20"
              },
              {
                "doc_count": 3179,
                "avg_apdexscore": {
                  "value": 0.8579545454545454
                },
                "key": "service21"
              },
              {
                "doc_count": 2190,
                "avg_apdexscore": {
                  "value": 0.8739224137931034
                },
                "key": "service24"
              }
            ]
          }
        },
        "slack": {
          "message": {
            "from": "_inlined_",
            "text": " service.name=service2 is below 0.9 with labels.apdexscore=0.7985335195530726 \n service.name=service3 is below 0.9 with labels.apdexscore=0.8258392362180474 \n service.name=service4 is below 0.9 with labels.apdexscore=0.7917981072555205 \n service.name=service7 is below 0.9 with labels.apdexscore=0.8404023926046764 \n service.name=service9 is below 0.9 with labels.apdexscore=0.8571428571428571 \n service.name=service10 is below 0.9 with labels.apdexscore=0.8769911504424779 \n service.name=service16 is below 0.9 with labels.apdexscore=0.8183503243744208 \n service.name=service17 is below 0.9 with labels.apdexscore=0.8734817813765182 \n service.name=service20 is below 0.9 with labels.apdexscore=0.867335562987737 \n service.name=service21 is below 0.9 with labels.apdexscore=0.8579545454545454 \n service.name=service24 is below 0.9 with labels.apdexscore=0.8739224137931034 \n"
          }
        }
      }
    ]
  },
  "messages": []
}

The simulate is working and sends the notifications to an slack channel. But when I save it I don't receive more notifications. Why could it be?

Kind regards

That does not make any sense whatsoever. You can have more than one action, so you could both log and slack. You could check the log (elasticsearch.log) so see if you are publishing information there when the watch is running. If it does, then it must be something wrong with the slack output.

I know it doesn't make sense. I included log along with slack action but it doesn't work. It just works when I do a simulation. :man_shrugging:

I include some screenshots from the watcher.