How to check some condition through all docs in aggregation

Hey ES community!

I have a query: (the part of code is below...btw script was taken from visualization in kibana request)

"aggs": {
         "uuid": {
         "terms": {
        "field": "device_uuid",
        "size": 5000,
        "order": {
          "_term": "desc"
        },
        "min_doc_count": "10"
        },
        "aggs": {
        "geo": {
          "cardinality": {
            "field": "location"
          }
        },
        "acc": {
          "cardinality": {
            "field": "accuracy"
          }
        },
        "age": {
          "cardinality": {
            "script": {
              "inline": "(doc['@timestamp'].date.getMillis() - doc['user_creation_time'].date.getMillis())/1000/60/60/24",
              "lang": "painless"
            }
          }
        }
      }
    }
  },
  "stored_fields": [
    "*"
  ],
  "script_fields": {
    "user_age": {
      "script": {
        "inline": "(doc['@timestamp'].date.getMillis() - doc['user_creation_time'].date.getMillis())/1000/60/60/24",
        "lang": "painless"
      }
    }
  },
  "docvalue_fields": [
    "@timestamp",
    "time",
    "user_creation_time"
  ],
  "query": {
    "bool": {
      "must": [
        {
          "match_all": {}
        },
        {
          "match_phrase": {
            "gc": {
              "query": 1
            }
          }
        },
        {
          "match_phrase": {
            "solution": {
              "query": "first"
            }
          }
        },

I want to create an alert: when I have geo = 1 and acc = 1 and age >=2_ notify me via slack.

But i cannot understand how to check stats in all output buckets. The output of watcher is (only part of output):

    "aggregations": {
              "uuid": {
                "doc_count_error_upper_bound": 0,
                "sum_other_doc_count": 411288,
                "buckets": [
                  {
                    "acc": {
                      "value": 37
                    },
                    "geo": {
                      "value": 42
                    },
                    "doc_count": 42,
                    "key": "234134",
                    "age": {
                      "value": 8
                    }
                  },
                  {
                    "acc": {
                      "value": 22
                    },
                    "geo": {
                      "value": 22
                    },
                    "doc_count": 25,
                    "key": "12342456",
                    "age": {
                      "value": 6
                    }
                  } 
.......

I only know how to check in first bucket:

ctx.payload.aggregations.uuid.buckets.0.acc.value

I want to put condition something like:

"if (ctx.payload.hits.total == 0) return false; def nb=(ctx.payload.aggregations.uuid.buckets.0.age.value); def nu = (ctx.payload.aggregations.uuid.buckets.0.acc.value); def ng=(ctx.payload.aggregations.uuid.buckets.0.geo.value); if (nb>=2 && nu==1 && ng==1) return true; ---btw it doesnt work even for first bucket (

Hello,

Any advice here.

Thanks in advance!

Hey,

I've tried this condition and it works (but only check in first bucket of docs :frowning: )

"condition": {
    "script": {
      "source": "if (ctx.payload.hits.total == 0) return false; def nb=(ctx.payload.aggregations.uuid.buckets.0.age.value); def nu = (ctx.payload.aggregations.uuid.buckets.0.acc.value); def ng=(ctx.payload.aggregations.uuid.buckets.0.geo.value); if (nb>=2 && nu==1 && ng==1) return true; return false",
      "lang": "painless"
    }
  },

How I can refer to all (so watcher will check through all docs in buckets not only first one)?

Hey community,

Can anyone give some advice here ?

Really appreciate any helpful words!

@spinscale, Hello Alex, sorry for disturbing, but can you advise on this topic please.

Much thanks in advance.

So basically i don't understand how to check condition taht i want to check in every docs in buckets, but not only as I mentioned (as I know) in first one (using buckets.0)

Hey,

take a look at this example

return ctx.payload.aggregations.uuid.buckets.stream().allMatch(b -> b.age.value > 5)

this would check for all the age values being greater than five. You can do more complex checks of course. Maybe also check the alerting examples in the examples repo

Hope this helps as a start.

--Alex

Hello Alex,
Much thanks for input!

Honestly don't understand how to use your example... ( I tried to use as below, but not work (I took numbers from bucket so it must be alerting..)

"condition": {
    "script": {
      "source": "if (ctx.payload.aggregations.uuid.buckets.stream().allMatch(b -> b.age.value >=4)&&ctx.payload.aggregations.uuid.buckets.stream().allMatch(d -> d.geo.value == 33)&&ctx.payload.aggregations.uuid.buckets.stream().allMatch(d -> d.acc.value == 16)) return true; return false",
      "lang": "painless"

I also tried this: also no result (but i see docs with age =4 in output):

"condition": {
        "script": {
          "source": "if (ctx.payload.aggregations.uuid.buckets.stream().allMatch(b -> b.age.value >=4) return true; return false",

or this:

"condition": {
    "script": {
      "source": "return ctx.payload.aggregations.uuid.buckets.stream().allMatch(b -> b.age.value >=4) ",
      "lang": "painless"

hey,

can you share the execute watch API output? The last one of your three examples looks good to me.

--Alex

Sure, here it is: (starting from part result)

          "hits": {
          "hits": [],
          "total": 53615,
          "max_score": 0
        },
        "took": 644,
        "timed_out": false,
        "aggregations": {
          "uuid": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 53076,
            "buckets": [
              {
                "acc": {
                  "value": 12
                },
                "geo": {
                  "value": 1
                },
                "doc_count": 13,
                "key": "ffde406be02787df",
                "age": {
                  "value": 2
                }
              },
              {
                "acc": {
                  "value": 8
                },
                "geo": {
                  "value": 11
                },
                "doc_count": 11,
                "key": "ffa3ebcd45173ec6",
                "age": {
                  "value": 2
                }
              },
              {
                "acc": {
                  "value": 15
                },
                "geo": {
                  "value": 15
                },
                "doc_count": 15,
                "key": "ff25efab59928c22",
                "age": {
                  "value": 2
                }
              },
              {
                "acc": {
                  "value": 49
                },
                "geo": {
                  "value": 47
                },
                "doc_count": 54,
                "key": "ff1acb37e1210841",
                "age": {
                  "value": 1
                }
              },
              {
                "acc": {
                  "value": 30
                },
                "geo": {
                  "value": 30
                },
                "doc_count": 32,
                "key": "fef9071210fa6d0b",
                "age": {
                  "value": 2
                }
              },
              {
                "acc": {
                  "value": 12
                },
                "geo": {
                  "value": 20
                },
                "doc_count": 20,
                "key": "fe10bbb1c0c023e7",
                "age": {
                  "value": 2
                }
              },
              {
                "acc": {
                  "value": 10
                },
                "geo": {
                  "value": 10
                },
                "doc_count": 10,
                "key": "fdc63ad5eb53a0",
                "age": {
                  "value": 1
                }
              },
              {
                "acc": {
                  "value": 9
                },
                "geo": {
                  "value": 16
                },
                "doc_count": 16,
                "key": "fd919e220793ee17",
                "age": {
                  "value": 3
                }
              }
            ]
          }
        }
      },
      "search": {
        "request": {
          "search_type": "query_then_fetch",
          "indices": [
            "indice_name*"
          ],
          "types": [
            "events"
          ],
          "body": {
            "size": 0,
            "_source": {
              "excludes": []
            },
            "aggs": {
              "uuid": {
                "terms": {
                  "field": "device_uuid",
                  "size": 10,
                  "order": {
                    "_term": "desc"
                  },
                  "min_doc_count": "10"
                },
                "aggs": {
                  "geo": {
                    "cardinality": {
                      "field": "best_geo"
                    }
                  },
                  "acc": {
                    "cardinality": {
                      "field": "best_accuracy"
                    }
                  },
                  "age": {
                    "cardinality": {
                      "script": {
                        "inline": "(doc['@timestamp'].date.getMillis() - doc['user_creation_time'].date.getMillis())/1000/60/60/24",
                        "lang": "painless"
                      }
                    }
                  }
                }
              }
            },
            "stored_fields": [
              "*"
            ],
            "script_fields": {
              "user_age": {
                "script": {
                  "inline": "(doc['@timestamp'].date.getMillis() - doc['user_creation_time'].date.getMillis())/1000/60/60/24",
                  "lang": "painless"
                }
              }
            },
            "docvalue_fields": [
              "@timestamp",
              "time",
              "user_creation_time"
            ],
            "query": {
              "bool": {
                "must": [
                  {
                    "match_all": {}
                  },
                  {
                    "match_phrase": {
                      "gc": {
                        "query": 1
                      }
                    }
                  },
                  {
                    "match_phrase": {
                      "solution": {
                        "query": "first"
                      }
                    }
                  },
                  {
                    "range": {
                      "@timestamp": {
                        "gte": "now-1d",
                        "lte": "now"
                      }
                    }
                  }
                ],
                "filter": [],
                "should": [],
                "must_not": []
              }
            }
          }
        }
      }
    },
    "condition": {
      "type": "script",
      "status": "success",
      "met": false
    },
    "actions": []
  },
  "messages": []
}

I check what numbers I have in age in buckets doc, then put one of the number in condition for age, so to simulate to see that condition is met, but it says that met: false...

Btw, I tried to play with allmatch vs anymatch --> I tried to use this one:

" source": "return ctx.payload.aggregations.uuid.buckets.stream().anyMatch(b -> b.age.value == 3) ",

and it is alerting, But in text of notification in slack I using:

"{{#ctx.payload.aggregations.uuid.buckets}}{{key}}: {{geo.value}} {{/ctx.payload.aggregations.uuid.buckets}}" - it shows all docs in bucket, not only doc where age =3 (may be it is correct and I used incorrectly it...)

@spinscale, Hey Alex, have any thoughts on this, please?
Much appreciate your input!

Be advised that this is a community forum and there are no service level agreements on replies and how long it takes, if at all.

If you need more timely support, Elastic is offering support subscriptions for exactly this.

Hello, Sorry for my impatience.

I want to share some info that I managed to do here:

I decided to use chain input here: in first I check only aggs for geo and use bucket_filter =1 to show only docs where geo=1. In the second input I check only aggs for acc with the same bucket_filter =1 to show only docs where acc=1. And in the second input the same procedure for aggs for age but with >=2.

But my question is: if it possible to take user_name from first output to check if this name is presented in second output and if yes to check this name if it is presented in third one - then notify me if name is presented in every output.

Hey,

I just used this condition:

condition": {
    "script": {
      "source": "if (ctx.payload.first.aggregations.uuid.key == ctx.payload.second.aggregations.uuid.key == ctx.payload.third.aggregations.uuid.key) return true; return false",
      "lang": "painless"    } 

Did I cover all things, so every user_name (key) from first are comparing with second output and with third output ?

You cannot compare three arrays like that. This script will not work at all. You need to gather the all keys for each search in a list and then can compare them with each other.

Can you share info how to do your advice, please

But my compare script seems work, at least when I compare two outputs: where I know that names are intersected it alerts, and where not - it doesn't alert. (I generate all keys and then check it in excel - to double check).

UPDATE: oh. really don't work my script, just simulate when 1 name is presented in every output but watcher doesn't alert

Thanks

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.