Hi,
We have created a lens visualization for showing the apdex score of APM traces and we would like to create an alert based on that score. Our knowledge in alerting and monitoring is still limited and we haven't found a solution to transform that in an alert.
I attach an image of the visualization:
And here is some extra info from the visualization:
If the score goes under 0.8 then it should trigger an alert.
Let mw know if you need any extra info. Thanks in advance.
Mario
Hi Mario, thanks for reaching out. We have this on our list of functionality to add. We are just about to release "Search threshold alerts" in Discover in 8.3 and, beyond that, this style of threshold alert based on a metric is absolutely where we want to go. Here's the issue for this improvement so you can follow along - [Lens] Alerting from Lens Visualizations · Issue #71150 · elastic/kibana · GitHub
Hi @ghudgins
Thanks for your quick answer.
Is there a way to extract the query info of the visualization so we can set an alert while we wait for 8.3 version? When should this new version be released?
Kind regards
Mario
The query for any visualization can be seen by using the "Inspect" option in Lens. Some visualizations require multiple queries to build the final end product--you'll be able to navigate the requests used to retrieve that metric.
Here is a full list of our alerting functionality today in 8.2 - Alerting | Kibana Guide [8.2] | Elastic
The search threshold coming in 8.3 (which is due out in the next few weeks) likely does NOT satisfy your use case but it's a step in the right direction.
We definitely have it on our roadmap to address the "Alerting from Lens Visualizations" issue I linked above but don't have a concrete timeframe I can commit to here at this time.
You could use Watcher in the meantime to create this alert, per service.name
Here's a similar example (using different data). This is responsetime per airline visualization:
And here's the equivalent watch (you would need to change this to not use the simulate (_execute) endpoint and configure whatever appropriate action you want:
POST _watcher/watch/_execute
{
"watch": {
"trigger": {
"schedule": {
"interval": "5m"
}
},
"input": {
"search": {
"request": {
"indices": [
"farequote"
],
"body": {
"size": 0,
"query": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"gte": "now-5m"
}
}
}
]
}
},
"aggs": {
"airlines": {
"terms": {
"field": "airline",
"size": 20
},
"aggs": {
"max_responsetime": {
"max": {
"field": "responsetime"
}
}
}
}
}
}
}
}
},
"condition": {
"script": """
return ctx.payload.aggregations.airlines.buckets.stream()
.filter(airlines -> airlines.max_responsetime.value > 3000)
.count() > 0
"""
},
"actions": {
"log": {
"transform": {
"script": """
return ctx.payload.aggregations.airlines.buckets.stream()
.filter(airlines -> airlines.max_responsetime.value > 3000)
.collect(Collectors.toList());
"""
},
"logging": {
"text": """
{{#ctx.payload._value}}
airline={{key}} exceeded 3sec threshold with responsetime={{max_responsetime.value}}
{{/ctx.payload._value}}
"""
}
}
}
}
}
Sample result would look like:
airline=SWR exceeded 3sec threshold with responsetime=4038.84
airline=NKS exceeded 3sec threshold with responsetime=10334.94
Thanks @richcollier!
We will try to use watcher in the meantime. If we succeed we will let you know 
Best regards
Mario
Hi @richcollier
I made some changes to your watcher example but I think something seems to be failing.
This is my _watcher post
POST _watcher/watch/_execute
{
"watch": {
"trigger": {
"schedule": {
"interval": "5m"
}
},
"input": {
"search": {
"request": {
"indices": [
"traces-apm*"
],
"body": {
"size": 0,
"query": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"gte": "now-5m"
}
}
}
]
}
},
"aggs": {
"services": {
"terms": {
"field": "service.name",
"size": 20
},
"aggs": {
"avg_apdexscore": {
"avg": {
"field": "labels.apdexscore"
}
}
}
}
}
}
}
}
},
"condition": {
"script": """
return ctx.payload.aggregations.services.buckets.stream()
.filter(services -> services.avg_apdexscore.value < 0.8)
.count() > 0
"""
},
"actions": {
"log": {
"transform": {
"script": """
return ctx.payload.aggregations.services.buckets.stream()
.filter(services -> services.avg_apdexscore.value > 3000)
.collect(Collectors.toList());
"""
},
"logging": {
"text": """
{{#ctx.payload._value}}
service.name={{key}} exceeded 3sec threshold with labels.apdexscore={{avg_apdexscore.value}}
{{/ctx.payload._value}}
"""
}
}
}
}
}
And this is the result
{
"_id" : "_inlined__**************",
"watch_record" : {
"watch_id" : "_inlined_",
"node" : "**************",
"state" : "failed",
"@timestamp" : "2022-06-21T10:44:26.439018514Z",
"user" : "**************",
"status" : {
"state" : {
"active" : true,
"timestamp" : "2022-06-21T10:44:26.438Z"
},
"actions" : {
"log" : {
"ack" : {
"timestamp" : "2022-06-21T10:44:26.438Z",
"state" : "awaits_successful_execution"
}
}
},
"execution_state" : "failed",
"version" : -1
},
"trigger_event" : {
"type" : "manual",
"triggered_time" : "2022-06-21T10:44:26.439Z",
"manual" : {
"schedule" : {
"scheduled_time" : "2022-06-21T10:44:26.439Z"
}
}
},
"input" : {
"search" : {
"request" : {
"search_type" : "query_then_fetch",
"indices" : [
"traces-apm*"
],
"rest_total_hits_as_int" : true,
"body" : {
"size" : 0,
"query" : {
"bool" : {
"filter" : [
{
"range" : {
"@timestamp" : {
"gte" : "now-5m"
}
}
}
]
}
},
"aggs" : {
"services" : {
"terms" : {
"field" : "service.name",
"size" : 20
},
"aggs" : {
"avg_apdexscore" : {
"avg" : {
"field" : "labels.apdexscore"
}
}
}
}
}
}
}
}
},
"condition" : {
"script" : {
"source" : """
return ctx.payload.aggregations.services.buckets.stream()
.filter(services -> services.avg_apdexscore.value < 0.8)
.count() > 0
""",
"lang" : "painless"
}
},
"result" : {
"execution_time" : "2022-06-21T10:44:26.439Z",
"execution_duration" : 3,
"input" : {
"type" : "search",
"status" : "success",
"payload" : {
"_shards" : {
"total" : 12,
"failed" : 0,
"successful" : 12,
"skipped" : 0
},
"hits" : {
"hits" : [ ],
"total" : 60,
"max_score" : null
},
"took" : 2,
"timed_out" : false,
"aggregations" : {
"services" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"doc_count" : 60,
"avg_apdexscore" : {
"value" : null
},
"key" : "staging-jucy"
}
]
}
}
},
"search" : {
"request" : {
"search_type" : "query_then_fetch",
"indices" : [
"traces-apm*"
],
"rest_total_hits_as_int" : true,
"body" : {
"size" : 0,
"query" : {
"bool" : {
"filter" : [
{
"range" : {
"@timestamp" : {
"gte" : "now-5m"
}
}
}
]
}
},
"aggs" : {
"services" : {
"terms" : {
"field" : "service.name",
"size" : 20
},
"aggs" : {
"avg_apdexscore" : {
"avg" : {
"field" : "labels.apdexscore"
}
}
}
}
}
}
}
}
},
"actions" : [ ]
},
"exception" : {
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"""services -> services.avg_apdexscore.value < 0.8)
.count() > 0
""",
" ^---- HERE"
],
"script" : " ...",
"lang" : "painless",
"position" : {
"offset" : 128,
"start" : 93,
"end" : 189
},
"caused_by" : {
"type" : "null_pointer_exception",
"reason" : """Cannot invoke "Object.getClass()" because "leftObject" is null""",
"stack_trace" : """java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "leftObject" is null
at org.elasticsearch.painless.DefBootstrap$MIC.checkLHS(DefBootstrap.java:416)
at org.elasticsearch.painless.PainlessScript$Script.lambda$synthetic$0( ...:129)
at org.elasticsearch.painless.PainlessScript$Script$$Lambda0.test(Unknown Source)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:178)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:258)
at java.base/java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:248)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.count(ReferencePipeline.java:709)
at org.elasticsearch.painless.PainlessScript$Script.execute( ...:168)
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.doExecute(ScriptCondition.java:65)
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.execute(ScriptCondition.java:60)
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:542)
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:342)
at org.elasticsearch.xpack.watcher.transport.actions.TransportExecuteWatchAction$1.doRun(TransportExecuteWatchAction.java:189)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:669)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
"""
},
"stack_trace" : """org.elasticsearch.script.ScriptException: runtime error
at org.elasticsearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:85)
at org.elasticsearch.painless.PainlessScript$Script.execute( ...:1)
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.doExecute(ScriptCondition.java:65)
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.execute(ScriptCondition.java:60)
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:542)
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:342)
at org.elasticsearch.xpack.watcher.transport.actions.TransportExecuteWatchAction$1.doRun(TransportExecuteWatchAction.java:189)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:669)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "leftObject" is null
at org.elasticsearch.painless.DefBootstrap$MIC.checkLHS(DefBootstrap.java:416)
at org.elasticsearch.painless.PainlessScript$Script.lambda$synthetic$0( ...:129)
at org.elasticsearch.painless.PainlessScript$Script$$Lambda0.test(Unknown Source)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:178)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:258)
at java.base/java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:248)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.count(ReferencePipeline.java:709)
at org.elasticsearch.painless.PainlessScript$Script.execute( ...:168)
... 11 more
"""
}
}
}
Not sure if there is something missing or I ddi something wrong
Kind Regards
Mario
Let's take Watcher out of the picture for a minute. What does the output look like for the query:
GET traces-apm*/_search
{
"size": 0,
"aggs": {
"services": {
"terms": {
"field": "service.name",
"size": 20
},
"aggs": {
"avg_apdexscore": {
"avg": {
"field": "labels.apdexscore"
}
}
}
}
}
}
I did run it in the wrong deployment 
Sorry about that!!
I did it now in the right place and it looks much better but still with some errors. The last query you sent works 
This is the result of the watcher post:
{
"_id" : "_inlined__*************",
"watch_record" : {
"watch_id" : "_inlined_",
"node" : "*************",
"state" : "failed",
"@timestamp" : "2022-06-21T14:56:49.136247915Z",
"user" : "*************",
"status" : {
"state" : {
"active" : true,
"timestamp" : "2022-06-21T14:56:49.128Z"
},
"actions" : {
"log" : {
"ack" : {
"timestamp" : "2022-06-21T14:56:49.128Z",
"state" : "awaits_successful_execution"
}
}
},
"execution_state" : "failed",
"version" : -1
},
"trigger_event" : {
"type" : "manual",
"triggered_time" : "2022-06-21T14:56:49.136Z",
"manual" : {
"schedule" : {
"scheduled_time" : "2022-06-21T14:56:49.136Z"
}
}
},
"input" : {
"search" : {
"request" : {
"search_type" : "query_then_fetch",
"indices" : [
"traces-apm*"
],
"rest_total_hits_as_int" : true,
"body" : {
"size" : 0,
"query" : {
"bool" : {
"filter" : [
{
"range" : {
"@timestamp" : {
"gte" : "now-5m"
}
}
}
]
}
},
"aggs" : {
"services" : {
"terms" : {
"field" : "service.name",
"size" : 20
},
"aggs" : {
"avg_apdexscore" : {
"avg" : {
"field" : "labels.apdexscore"
}
}
}
}
}
}
}
}
},
"condition" : {
"script" : {
"source" : """
return ctx.payload.aggregations.services.buckets.stream()
.filter(services -> services.avg_apdexscore.value < 0.8)
.count() > 0
""",
"lang" : "painless"
}
},
"result" : {
"execution_time" : "2022-06-21T14:56:49.136Z",
"execution_duration" : 1279,
"input" : {
"type" : "search",
"status" : "success",
"payload" : {
"_shards" : {
"total" : 51,
"failed" : 0,
"successful" : 51,
"skipped" : 0
},
"hits" : {
"hits" : [ ],
"total" : 10000,
"max_score" : null
},
"took" : 1269,
"timed_out" : false,
"aggregations" : {
"services" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 6420,
"buckets" : [
{
"doc_count" : 120052,
"avg_apdexscore" : {
"value" : 0.9906687402799378
},
"key" : "service1"
},
{
"doc_count" : 61031,
"avg_apdexscore" : {
"value" : null
},
"key" : "service2"
},
{
"doc_count" : 36053,
"avg_apdexscore" : {
"value" : 0.9924242424242424
},
"key" : "service3"
},
{
"doc_count" : 33921,
"avg_apdexscore" : {
"value" : 0.9965166908563134
},
"key" : "service4"
},
{
"doc_count" : 28276,
"avg_apdexscore" : {
"value" : 0.9984508133230054
},
"key" : "service5"
},
{
"doc_count" : 22541,
"avg_apdexscore" : {
"value" : 0.9969179706021811
},
"key" : "service6"
},
{
"doc_count" : 20524,
"avg_apdexscore" : {
"value" : 0.9965139442231076
},
"key" : "service7"
},
{
"doc_count" : 17962,
"avg_apdexscore" : {
"value" : null
},
"key" : "service8"
},
{
"doc_count" : 14103,
"avg_apdexscore" : {
"value" : null
},
"key" : "service9"
},
{
"doc_count" : 13839,
"avg_apdexscore" : {
"value" : 0.9953401677539608
},
"key" : "service10"
},
{
"doc_count" : 13617,
"avg_apdexscore" : {
"value" : 0.9990341274951706
},
"key" : "service11"
},
{
"doc_count" : 12318,
"avg_apdexscore" : {
"value" : null
},
"key" : "service12"
},
{
"doc_count" : 12066,
"avg_apdexscore" : {
"value" : 0.9987725040916531
},
"key" : "service13"
},
{
"doc_count" : 10642,
"avg_apdexscore" : {
"value" : null
},
"key" : "service14"
},
{
"doc_count" : 9688,
"avg_apdexscore" : {
"value" : 0.9993946731234867
},
"key" : "service15"
},
{
"doc_count" : 8857,
"avg_apdexscore" : {
"value" : null
},
"key" : "service16"
},
{
"doc_count" : 5326,
"avg_apdexscore" : {
"value" : null
},
"key" : "service17"
},
{
"doc_count" : 5322,
"avg_apdexscore" : {
"value" : null
},
"key" : "service18"
},
{
"doc_count" : 5156,
"avg_apdexscore" : {
"value" : null
},
"key" : "service19"
},
{
"doc_count" : 3863,
"avg_apdexscore" : {
"value" : null
},
"key" : "service20"
}
]
}
}
},
"search" : {
"request" : {
"search_type" : "query_then_fetch",
"indices" : [
"traces-apm*"
],
"rest_total_hits_as_int" : true,
"body" : {
"size" : 0,
"query" : {
"bool" : {
"filter" : [
{
"range" : {
"@timestamp" : {
"gte" : "now-5m"
}
}
}
]
}
},
"aggs" : {
"services" : {
"terms" : {
"field" : "service.name",
"size" : 20
},
"aggs" : {
"avg_apdexscore" : {
"avg" : {
"field" : "labels.apdexscore"
}
}
}
}
}
}
}
}
},
"actions" : [ ]
},
"exception" : {
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"""services -> services.avg_apdexscore.value < 0.8)
.count() > 0
""",
" ^---- HERE"
],
"script" : " ...",
"lang" : "painless",
"position" : {
"offset" : 128,
"start" : 93,
"end" : 189
},
"caused_by" : {
"type" : "null_pointer_exception",
"reason" : """Cannot invoke "Object.getClass()" because "leftObject" is null""",
"stack_trace" : """java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "leftObject" is null
at org.elasticsearch.painless.DefBootstrap$MIC.checkLHS(DefBootstrap.java:416)
at org.elasticsearch.painless.PainlessScript$Script.lambda$synthetic$0( ...:129)
at org.elasticsearch.painless.PainlessScript$Script$$Lambda0.test(Unknown Source)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:178)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:258)
at java.base/java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:248)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.count(ReferencePipeline.java:709)
at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:732)
at org.elasticsearch.painless.DefBootstrap$PIC.fallback(DefBootstrap.java:239)
at org.elasticsearch.painless.PainlessScript$Script.execute( ...:168)
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.doExecute(ScriptCondition.java:65)
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.execute(ScriptCondition.java:60)
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:542)
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:342)
at org.elasticsearch.xpack.watcher.transport.actions.TransportExecuteWatchAction$1.doRun(TransportExecuteWatchAction.java:189)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:669)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
"""
},
"stack_trace" : """org.elasticsearch.script.ScriptException: runtime error
at org.elasticsearch.painless.PainlessScript.convertToScriptException(PainlessScript.java:85)
at org.elasticsearch.painless.PainlessScript$Script.execute( ...:1)
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.doExecute(ScriptCondition.java:65)
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.execute(ScriptCondition.java:60)
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:542)
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:342)
at org.elasticsearch.xpack.watcher.transport.actions.TransportExecuteWatchAction$1.doRun(TransportExecuteWatchAction.java:189)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:669)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "leftObject" is null
at org.elasticsearch.painless.DefBootstrap$MIC.checkLHS(DefBootstrap.java:416)
at org.elasticsearch.painless.PainlessScript$Script.lambda$synthetic$0( ...:129)
at org.elasticsearch.painless.PainlessScript$Script$$Lambda0.test(Unknown Source)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:178)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:258)
at java.base/java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:248)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.count(ReferencePipeline.java:709)
at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:732)
at org.elasticsearch.painless.DefBootstrap$PIC.fallback(DefBootstrap.java:239)
at org.elasticsearch.painless.PainlessScript$Script.execute( ...:168)
... 11 more
"""
}
}
}
Thanks
In case you also need the query result here it is:
{
"took" : 8327,
"timed_out" : false,
"_shards" : {
"total" : 50,
"successful" : 50,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 10000,
"relation" : "gte"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"services" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 37060702,
"buckets" : [
{
"key" : "service1",
"doc_count" : 751728057,
"avg_apdexscore" : {
"value" : 0.9900452781045794
}
},
{
"key" : "service2",
"doc_count" : 368399400,
"avg_apdexscore" : {
"value" : null
}
},
{
"key" : "service3",
"doc_count" : 187379449,
"avg_apdexscore" : {
"value" : 0.9970768098569803
}
},
{
"key" : "service4",
"doc_count" : 166119692,
"avg_apdexscore" : {
"value" : 0.9981641091946719
}
},
{
"key" : "service5",
"doc_count" : 140902539,
"avg_apdexscore" : {
"value" : 0.9905411018902832
}
},
{
"key" : "service6",
"doc_count" : 131635318,
"avg_apdexscore" : {
"value" : 0.9977093144852212
}
},
{
"key" : "service7",
"doc_count" : 113643554,
"avg_apdexscore" : {
"value" : 0.9985224096879122
}
},
{
"key" : "service8",
"doc_count" : 98480453,
"avg_apdexscore" : {
"value" : 0.998956908884416
}
},
{
"key" : "service9",
"doc_count" : 97515652,
"avg_apdexscore" : {
"value" : null
}
},
{
"key" : "service10",
"doc_count" : 84340547,
"avg_apdexscore" : {
"value" : 0.9955444573071522
}
},
{
"key" : "service11",
"doc_count" : 74934287,
"avg_apdexscore" : {
"value" : null
}
},
{
"key" : "service12",
"doc_count" : 72040453,
"avg_apdexscore" : {
"value" : 0.99558658505802
}
},
{
"key" : "service13",
"doc_count" : 69954389,
"avg_apdexscore" : {
"value" : 0.9972449236072635
}
},
{
"key" : "service14",
"doc_count" : 63052293,
"avg_apdexscore" : {
"value" : null
}
},
{
"key" : "service15",
"doc_count" : 54924435,
"avg_apdexscore" : {
"value" : null
}
},
{
"key" : "service16",
"doc_count" : 51429637,
"avg_apdexscore" : {
"value" : null
}
},
{
"key" : "service17",
"doc_count" : 41927901,
"avg_apdexscore" : {
"value" : null
}
},
{
"key" : "service18",
"doc_count" : 41654399,
"avg_apdexscore" : {
"value" : null
}
},
{
"key" : "service19",
"doc_count" : 32813611,
"avg_apdexscore" : {
"value" : null
}
},
{
"key" : "service20",
"doc_count" : 29704495,
"avg_apdexscore" : {
"value" : 0.9965565582521134
}
}
]
}
}
}
Your problem is that some of your services have null data:
If this is expected, we'll just need to modify the code to protect against the null comparison...
Hi @richcollier
I was able to fix the null values. There was a missing processor script for one of the ingest pipelines of the apm indices.
{
"_id" : "_inlined__**************",
"watch_record" : {
"watch_id" : "_inlined_",
"node" : "**************",
"state" : "executed",
"@timestamp" : "2022-06-22T07:36:53.566141447Z",
"user" : "**************",
"status" : {
"state" : {
"active" : true,
"timestamp" : "2022-06-22T07:36:53.565Z"
},
"last_checked" : "2022-06-22T07:36:53.566Z",
"last_met_condition" : "2022-06-22T07:36:53.566Z",
"actions" : {
"log" : {
"ack" : {
"timestamp" : "2022-06-22T07:36:53.566Z",
"state" : "ackable"
},
"last_execution" : {
"timestamp" : "2022-06-22T07:36:53.566Z",
"successful" : true
},
"last_successful_execution" : {
"timestamp" : "2022-06-22T07:36:53.566Z",
"successful" : true
}
}
},
"execution_state" : "executed",
"version" : -1
},
"trigger_event" : {
"type" : "manual",
"triggered_time" : "2022-06-22T07:36:53.566Z",
"manual" : {
"schedule" : {
"scheduled_time" : "2022-06-22T07:36:53.566Z"
}
}
},
"input" : {
"search" : {
"request" : {
"search_type" : "query_then_fetch",
"indices" : [
"traces-apm*"
],
"rest_total_hits_as_int" : true,
"body" : {
"size" : 0,
"query" : {
"bool" : {
"filter" : [
{
"range" : {
"@timestamp" : {
"gte" : "now-5m"
}
}
}
]
}
},
"aggs" : {
"services" : {
"terms" : {
"field" : "service.name",
"size" : 30
},
"aggs" : {
"avg_apdexscore" : {
"avg" : {
"field" : "labels.apdexscore"
}
}
}
}
}
}
}
}
},
"condition" : {
"script" : {
"source" : """
return ctx.payload.aggregations.services.buckets.stream()
.filter(services -> services.avg_apdexscore.value < 0.8)
.count() > 0
""",
"lang" : "painless"
}
},
"result" : {
"execution_time" : "2022-06-22T07:36:53.566Z",
"execution_duration" : 68,
"input" : {
"type" : "search",
"status" : "success",
"payload" : {
"_shards" : {
"total" : 50,
"failed" : 0,
"successful" : 50,
"skipped" : 0
},
"hits" : {
"hits" : [ ],
"total" : 10000,
"max_score" : null
},
"took" : 66,
"timed_out" : false,
"aggregations" : {
"services" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"doc_count" : 99528,
"avg_apdexscore" : {
"value" : 0.991608046402567
},
"key" : "service1"
},
{
"doc_count" : 52511,
"avg_apdexscore" : {
"value" : 0.8421935102081073
},
"key" : "service2"
},
{
"doc_count" : 28721,
"avg_apdexscore" : {
"value" : 0.9943089430894309
},
"key" : "service3"
},
{
"doc_count" : 25580,
"avg_apdexscore" : {
"value" : 0.9958333333333333
},
"key" : "service4"
},
{
"doc_count" : 25494,
"avg_apdexscore" : {
"value" : 0.9980323567993004
},
"key" : "service5"
},
{
"doc_count" : 15721,
"avg_apdexscore" : {
"value" : 1.0
},
"key" : "service6"
},
{
"doc_count" : 14718,
"avg_apdexscore" : {
"value" : 0.9984394506866417
},
"key" : "service7"
},
{
"doc_count" : 14430,
"avg_apdexscore" : {
"value" : 0.7919758812615956
},
"key" : "service8"
},
{
"doc_count" : 14377,
"avg_apdexscore" : {
"value" : 0.9991749174917491
},
"key" : "service9"
},
{
"doc_count" : 14250,
"avg_apdexscore" : {
"value" : 0.9990221642764016
},
"key" : "service10"
},
{
"doc_count" : 12484,
"avg_apdexscore" : {
"value" : 0.812205466540999
},
"key" : "service11"
},
{
"doc_count" : 10796,
"avg_apdexscore" : {
"value" : 0.8555351401010565
},
"key" : "service12"
},
{
"doc_count" : 9479,
"avg_apdexscore" : {
"value" : 0.9973375931842385
},
"key" : "service13"
},
{
"doc_count" : 8413,
"avg_apdexscore" : {
"value" : 0.9965277777777778
},
"key" : "service14"
},
{
"doc_count" : 7784,
"avg_apdexscore" : {
"value" : 0.8818503241013553
},
"key" : "service15"
},
{
"doc_count" : 7688,
"avg_apdexscore" : {
"value" : 0.8477975016436555
},
"key" : "service16"
},
{
"doc_count" : 6477,
"avg_apdexscore" : {
"value" : 0.8296387520525451
},
"key" : "service17"
},
{
"doc_count" : 5683,
"avg_apdexscore" : {
"value" : 0.8875106928999145
},
"key" : "service18"
},
{
"doc_count" : 4275,
"avg_apdexscore" : {
"value" : 0.9988317757009346
},
"key" : "service19"
},
{
"doc_count" : 3455,
"avg_apdexscore" : {
"value" : 0.8892005610098177
},
"key" : "service20"
},
{
"doc_count" : 2700,
"avg_apdexscore" : {
"value" : 0.8570205479452054
},
"key" : "service21"
},
{
"doc_count" : 2600,
"avg_apdexscore" : {
"value" : 0.8529411764705882
},
"key" : "service22"
},
{
"doc_count" : 278,
"avg_apdexscore" : {
"value" : 0.996
},
"key" : "service23"
},
{
"doc_count" : 30,
"avg_apdexscore" : {
"value" : 1.0
},
"key" : "service24"
}
]
}
}
},
"search" : {
"request" : {
"search_type" : "query_then_fetch",
"indices" : [
"traces-apm*"
],
"rest_total_hits_as_int" : true,
"body" : {
"size" : 0,
"query" : {
"bool" : {
"filter" : [
{
"range" : {
"@timestamp" : {
"gte" : "now-5m"
}
}
}
]
}
},
"aggs" : {
"services" : {
"terms" : {
"field" : "service.name",
"size" : 30
},
"aggs" : {
"avg_apdexscore" : {
"avg" : {
"field" : "labels.apdexscore"
}
}
}
}
}
}
}
}
},
"condition" : {
"type" : "script",
"status" : "success",
"met" : true
},
"actions" : [
{
"id" : "log",
"type" : "logging",
"status" : "success",
"transform" : {
"type" : "script",
"status" : "success",
"payload" : {
"_value" : [ ]
}
},
"logging" : {
"logged_text" : """
"""
}
}
]
},
"messages" : [ ]
}
}
To summarise, this calculates the average apdex score by service during the last 5 minutes, right?
Kind regards
Yes it does. I see that you're not getting any output in the logged_text and then I noticed that you probably have a typo in your action:
It should not be > 3000 (that was my example). It should be < 0.8
Hi @richcollier
I was able to create the watcher and simulate some notifications. I modified the actions to notify to slack when there is a value below 0.9 (It will be 0.8 once it works as expected).
Here is the JSON content of the watcher
{
"trigger": {
"schedule": {
"interval": "5m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"traces-apm*"
],
"rest_total_hits_as_int": true,
"body": {
"size": 0,
"query": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"gte": "now-5m"
}
}
}
]
}
},
"aggs": {
"services": {
"terms": {
"field": "service.name",
"size": 30
},
"aggs": {
"avg_apdexscore": {
"avg": {
"field": "labels.apdexscore"
}
}
}
}
}
}
}
}
},
"condition": {
"script": {
"source": """
return ctx.payload.aggregations.services.buckets.stream()
.filter(services -> services.avg_apdexscore.value < 0.9)
.count() > 0
""",
"lang": "painless"
}
},
"actions": {
"notify-slack": {
"throttle_period_in_millis": 300000,
"transform": {
"script": {
"source": """
return ctx.payload.aggregations.services.buckets.stream()
.filter(services -> services.avg_apdexscore.value < 0.9)
.collect(Collectors.toList());
""",
"lang": "painless"
}
},
"slack": {
"message": {
"text": """{{#ctx.payload._value}} service.name={{key}} is below 0.9 with labels.apdexscore={{avg_apdexscore.value}}
{{/ctx.payload._value}}"""
}
}
}
}
}
And this is a Simulate watch
{
"watch_id": "_inlined_",
"node": "**************",
"state": "executed",
"@timestamp": "2022-06-22T13:30:32.298895082Z",
"user": "**************",
"status": {
"state": {
"active": true,
"timestamp": "2022-06-22T13:30:32.298Z"
},
"last_checked": "2022-06-22T13:30:32.298Z",
"last_met_condition": "2022-06-22T13:30:32.298Z",
"actions": {
"notify-slack": {
"ack": {
"timestamp": "2022-06-22T13:30:32.298Z",
"state": "ackable"
},
"last_execution": {
"timestamp": "2022-06-22T13:30:32.298Z",
"successful": true
},
"last_successful_execution": {
"timestamp": "2022-06-22T13:30:32.298Z",
"successful": true
}
}
},
"execution_state": "executed",
"version": -1
},
"trigger_event": {
"type": "manual",
"triggered_time": "2022-06-22T13:30:32.298Z",
"manual": {
"schedule": {
"scheduled_time": "2022-06-22T13:30:32.298Z"
}
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"traces-apm*"
],
"rest_total_hits_as_int": true,
"body": {
"size": 0,
"query": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"gte": "now-5m"
}
}
}
]
}
},
"aggs": {
"services": {
"terms": {
"field": "service.name",
"size": 30
},
"aggs": {
"avg_apdexscore": {
"avg": {
"field": "labels.apdexscore"
}
}
}
}
}
}
}
}
},
"condition": {
"script": {
"source": "\n return ctx.payload.aggregations.services.buckets.stream() \n .filter(services -> services.avg_apdexscore.value < 0.9) \n .count() > 0 \n ",
"lang": "painless"
}
},
"metadata": {
"name": "Apdex score by service",
"xpack": {
"type": "json"
}
},
"result": {
"execution_time": "2022-06-22T13:30:32.298Z",
"execution_duration": 2952,
"input": {
"type": "search",
"status": "success",
"payload": {
"_shards": {
"total": 52,
"failed": 0,
"successful": 52,
"skipped": 0
},
"hits": {
"hits": [],
"total": 10000,
"max_score": null
},
"took": 2950,
"timed_out": false,
"aggregations": {
"services": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"doc_count": 124695,
"avg_apdexscore": {
"value": 0.9914766892585747
},
"key": "service1"
},
{
"doc_count": 65515,
"avg_apdexscore": {
"value": 0.7985335195530726
},
"key": "service2"
},
{
"doc_count": 38725,
"avg_apdexscore": {
"value": 0.9988324576765908
},
"key": "service3"
},
{
"doc_count": 34317,
"avg_apdexscore": {
"value": 0.9955449955449955
},
"key": "service4"
},
{
"doc_count": 22326,
"avg_apdexscore": {
"value": 0.9893672199170125
},
"key": "service5"
},
{
"doc_count": 19625,
"avg_apdexscore": {
"value": 0.9994541484716157
},
"key": "service6"
},
{
"doc_count": 18998,
"avg_apdexscore": {
"value": 0.9980136208853575
},
"key": "service7"
},
{
"doc_count": 18692,
"avg_apdexscore": {
"value": 0.8258392362180474
},
"key": "service8"
},
{
"doc_count": 17580,
"avg_apdexscore": {
"value": 0.7917981072555205
},
"key": "service9"
},
{
"doc_count": 12413,
"avg_apdexscore": {
"value": 0.9996461429582448
},
"key": "service10"
},
{
"doc_count": 10132,
"avg_apdexscore": {
"value": 0.9968916518650088
},
"key": "service11"
},
{
"doc_count": 9431,
"avg_apdexscore": {
"value": 0.9986376021798365
},
"key": "service12"
},
{
"doc_count": 9045,
"avg_apdexscore": {
"value": 0.8404023926046764
},
"key": "service13"
},
{
"doc_count": 8522,
"avg_apdexscore": {
"value": 0.9949109414758269
},
"key": "service14"
},
{
"doc_count": 8322,
"avg_apdexscore": {
"value": 0.8571428571428571
},
"key": "service15"
},
{
"doc_count": 7167,
"avg_apdexscore": {
"value": 0.8769911504424779
},
"key": "service16"
},
{
"doc_count": 5987,
"avg_apdexscore": {
"value": 0.8183503243744208
},
"key": "service17"
},
{
"doc_count": 4937,
"avg_apdexscore": {
"value": 0.8734817813765182
},
"key": "service18"
},
{
"doc_count": 4350,
"avg_apdexscore": {
"value": 0.867335562987737
},
"key": "service19"
},
{
"doc_count": 4080,
"avg_apdexscore": {
"value": 0.9984984984984985
},
"key": "service20"
},
{
"doc_count": 3179,
"avg_apdexscore": {
"value": 0.8579545454545454
},
"key": "service21"
},
{
"doc_count": 2190,
"avg_apdexscore": {
"value": 0.8739224137931034
},
"key": "service22"
},
{
"doc_count": 780,
"avg_apdexscore": {
"value": 0.991701244813278
},
"key": "service23"
},
{
"doc_count": 30,
"avg_apdexscore": {
"value": 1
},
"key": "service24"
}
]
}
}
},
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"traces-apm*"
],
"rest_total_hits_as_int": true,
"body": {
"size": 0,
"query": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"gte": "now-5m"
}
}
}
]
}
},
"aggs": {
"services": {
"terms": {
"field": "service.name",
"size": 30
},
"aggs": {
"avg_apdexscore": {
"avg": {
"field": "labels.apdexscore"
}
}
}
}
}
}
}
}
},
"condition": {
"type": "script",
"status": "success",
"met": true
},
"actions": [
{
"id": "notify-slack",
"type": "slack",
"status": "simulated",
"transform": {
"type": "script",
"status": "success",
"payload": {
"_value": [
{
"doc_count": 65515,
"avg_apdexscore": {
"value": 0.7985335195530726
},
"key": "service2"
},
{
"doc_count": 18692,
"avg_apdexscore": {
"value": 0.8258392362180474
},
"key": "service3"
},
{
"doc_count": 17580,
"avg_apdexscore": {
"value": 0.7917981072555205
},
"key": "service4"
},
{
"doc_count": 9045,
"avg_apdexscore": {
"value": 0.8404023926046764
},
"key": "service7"
},
{
"doc_count": 8322,
"avg_apdexscore": {
"value": 0.8571428571428571
},
"key": "service9"
},
{
"doc_count": 7167,
"avg_apdexscore": {
"value": 0.8769911504424779
},
"key": "service10"
},
{
"doc_count": 5987,
"avg_apdexscore": {
"value": 0.8183503243744208
},
"key": "service16"
},
{
"doc_count": 4937,
"avg_apdexscore": {
"value": 0.8734817813765182
},
"key": "service17"
},
{
"doc_count": 4350,
"avg_apdexscore": {
"value": 0.867335562987737
},
"key": "service20"
},
{
"doc_count": 3179,
"avg_apdexscore": {
"value": 0.8579545454545454
},
"key": "service21"
},
{
"doc_count": 2190,
"avg_apdexscore": {
"value": 0.8739224137931034
},
"key": "service24"
}
]
}
},
"slack": {
"message": {
"from": "_inlined_",
"text": " service.name=service2 is below 0.9 with labels.apdexscore=0.7985335195530726 \n service.name=service3 is below 0.9 with labels.apdexscore=0.8258392362180474 \n service.name=service4 is below 0.9 with labels.apdexscore=0.7917981072555205 \n service.name=service7 is below 0.9 with labels.apdexscore=0.8404023926046764 \n service.name=service9 is below 0.9 with labels.apdexscore=0.8571428571428571 \n service.name=service10 is below 0.9 with labels.apdexscore=0.8769911504424779 \n service.name=service16 is below 0.9 with labels.apdexscore=0.8183503243744208 \n service.name=service17 is below 0.9 with labels.apdexscore=0.8734817813765182 \n service.name=service20 is below 0.9 with labels.apdexscore=0.867335562987737 \n service.name=service21 is below 0.9 with labels.apdexscore=0.8579545454545454 \n service.name=service24 is below 0.9 with labels.apdexscore=0.8739224137931034 \n"
}
}
}
]
},
"messages": []
}
The simulate is working and sends the notifications to an slack channel. But when I save it I don't receive more notifications. Why could it be?
Kind regards
That does not make any sense whatsoever. You can have more than one action, so you could both log and slack. You could check the log (elasticsearch.log) so see if you are publishing information there when the watch is running. If it does, then it must be something wrong with the slack output.
I know it doesn't make sense. I included log along with slack action but it doesn't work. It just works when I do a simulation. 
I include some screenshots from the watcher.
Hi,
Elastic engineers couldn't find a solution. Could it be a bug?
Thanks
What version are you using?
This should be an easy thing to replicate to see if it is a bug...
escloud 8.1.2
Hi,
We finally found the bug and fixed it.
After enabling “DEBUG” logs for the watcher on the cluster we found the following error:
Jul 14, 2022 @ 08:39:02.434 [instance-0000000017] error validating to start watcher java.lang.IllegalStateException: Alias [.watcher-history-16] points to more than one index
This was well-known issue by elastic and already fixed in Elasticsearch versions 8.1.3+ and 8.2.0+ so after upgrading our current escloud version the watcher started working.
Mario
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.