How to check some condition through all docs in aggregation

_Sergey · July 9, 2018, 11:31am

Hey ES community!

I have a query: (the part of code is below...btw script was taken from visualization in kibana request)

"aggs": {
         "uuid": {
         "terms": {
        "field": "device_uuid",
        "size": 5000,
        "order": {
          "_term": "desc"
        },
        "min_doc_count": "10"
        },
        "aggs": {
        "geo": {
          "cardinality": {
            "field": "location"
          }
        },
        "acc": {
          "cardinality": {
            "field": "accuracy"
          }
        },
        "age": {
          "cardinality": {
            "script": {
              "inline": "(doc['@timestamp'].date.getMillis() - doc['user_creation_time'].date.getMillis())/1000/60/60/24",
              "lang": "painless"
            }
          }
        }
      }
    }
  },
  "stored_fields": [
    "*"
  ],
  "script_fields": {
    "user_age": {
      "script": {
        "inline": "(doc['@timestamp'].date.getMillis() - doc['user_creation_time'].date.getMillis())/1000/60/60/24",
        "lang": "painless"
      }
    }
  },
  "docvalue_fields": [
    "@timestamp",
    "time",
    "user_creation_time"
  ],
  "query": {
    "bool": {
      "must": [
        {
          "match_all": {}
        },
        {
          "match_phrase": {
            "gc": {
              "query": 1
            }
          }
        },
        {
          "match_phrase": {
            "solution": {
              "query": "first"
            }
          }
        },

I want to create an alert: when I have geo = 1 and acc = 1 and age >=2_ notify me via slack.

But i cannot understand how to check stats in all output buckets. The output of watcher is (only part of output):

    "aggregations": {
              "uuid": {
                "doc_count_error_upper_bound": 0,
                "sum_other_doc_count": 411288,
                "buckets": [
                  {
                    "acc": {
                      "value": 37
                    },
                    "geo": {
                      "value": 42
                    },
                    "doc_count": 42,
                    "key": "234134",
                    "age": {
                      "value": 8
                    }
                  },
                  {
                    "acc": {
                      "value": 22
                    },
                    "geo": {
                      "value": 22
                    },
                    "doc_count": 25,
                    "key": "12342456",
                    "age": {
                      "value": 6
                    }
                  } 
.......

I only know how to check in first bucket:

ctx.payload.aggregations.uuid.buckets.0.acc.value

I want to put condition something like:

"if (ctx.payload.hits.total == 0) return false; def nb=(ctx.payload.aggregations.uuid.buckets.0.age.value); def nu = (ctx.payload.aggregations.uuid.buckets.0.acc.value); def ng=(ctx.payload.aggregations.uuid.buckets.0.geo.value); if (nb>=2 && nu==1 && ng==1) return true; ---btw it doesnt work even for first bucket (

_Sergey · July 9, 2018, 2:30pm

Hello,

Any advice here.

Thanks in advance!

_Sergey · July 11, 2018, 12:32pm

Hey,

I've tried this condition and it works (but only check in first bucket of docs )

"condition": {
    "script": {
      "source": "if (ctx.payload.hits.total == 0) return false; def nb=(ctx.payload.aggregations.uuid.buckets.0.age.value); def nu = (ctx.payload.aggregations.uuid.buckets.0.acc.value); def ng=(ctx.payload.aggregations.uuid.buckets.0.geo.value); if (nb>=2 && nu==1 && ng==1) return true; return false",
      "lang": "painless"
    }
  },

How I can refer to all (so watcher will check through all docs in buckets not only first one)?

_Sergey · July 13, 2018, 3:42pm

Hey community,

Can anyone give some advice here ?

Really appreciate any helpful words!

_Sergey · July 13, 2018, 3:50pm

@spinscale, Hello Alex, sorry for disturbing, but can you advise on this topic please.

Much thanks in advance.

So basically i don't understand how to check condition taht i want to check in every docs in buckets, but not only as I mentioned (as I know) in first one (using buckets.0)

spinscale · July 17, 2018, 7:20am

Hey,

take a look at this example

return ctx.payload.aggregations.uuid.buckets.stream().allMatch(b -> b.age.value > 5)

this would check for all the age values being greater than five. You can do more complex checks of course. Maybe also check the alerting examples in the examples repo

Hope this helps as a start.

--Alex

_Sergey · July 17, 2018, 9:58am

Hello Alex,
Much thanks for input!

Honestly don't understand how to use your example... ( I tried to use as below, but not work (I took numbers from bucket so it must be alerting..)

"condition": {
    "script": {
      "source": "if (ctx.payload.aggregations.uuid.buckets.stream().allMatch(b -> b.age.value >=4)&&ctx.payload.aggregations.uuid.buckets.stream().allMatch(d -> d.geo.value == 33)&&ctx.payload.aggregations.uuid.buckets.stream().allMatch(d -> d.acc.value == 16)) return true; return false",
      "lang": "painless"

I also tried this: also no result (but i see docs with age =4 in output):

"condition": {
        "script": {
          "source": "if (ctx.payload.aggregations.uuid.buckets.stream().allMatch(b -> b.age.value >=4) return true; return false",

or this:

"condition": {
    "script": {
      "source": "return ctx.payload.aggregations.uuid.buckets.stream().allMatch(b -> b.age.value >=4) ",
      "lang": "painless"

spinscale · July 17, 2018, 10:14am

hey,

can you share the execute watch API output? The last one of your three examples looks good to me.

--Alex

_Sergey · July 17, 2018, 10:32am

Sure, here it is: (starting from part result)

          "hits": {
          "hits": [],
          "total": 53615,
          "max_score": 0
        },
        "took": 644,
        "timed_out": false,
        "aggregations": {
          "uuid": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 53076,
            "buckets": [
              {
                "acc": {
                  "value": 12
                },
                "geo": {
                  "value": 1
                },
                "doc_count": 13,
                "key": "ffde406be02787df",
                "age": {
                  "value": 2
                }
              },
              {
                "acc": {
                  "value": 8
                },
                "geo": {
                  "value": 11
                },
                "doc_count": 11,
                "key": "ffa3ebcd45173ec6",
                "age": {
                  "value": 2
                }
              },
              {
                "acc": {
                  "value": 15
                },
                "geo": {
                  "value": 15
                },
                "doc_count": 15,
                "key": "ff25efab59928c22",
                "age": {
                  "value": 2
                }
              },
              {
                "acc": {
                  "value": 49
                },
                "geo": {
                  "value": 47
                },
                "doc_count": 54,
                "key": "ff1acb37e1210841",
                "age": {
                  "value": 1
                }
              },
              {
                "acc": {
                  "value": 30
                },
                "geo": {
                  "value": 30
                },
                "doc_count": 32,
                "key": "fef9071210fa6d0b",
                "age": {
                  "value": 2
                }
              },
              {
                "acc": {
                  "value": 12
                },
                "geo": {
                  "value": 20
                },
                "doc_count": 20,
                "key": "fe10bbb1c0c023e7",
                "age": {
                  "value": 2
                }
              },
              {
                "acc": {
                  "value": 10
                },
                "geo": {
                  "value": 10
                },
                "doc_count": 10,
                "key": "fdc63ad5eb53a0",
                "age": {
                  "value": 1
                }
              },
              {
                "acc": {
                  "value": 9
                },
                "geo": {
                  "value": 16
                },
                "doc_count": 16,
                "key": "fd919e220793ee17",
                "age": {
                  "value": 3
                }
              }
            ]
          }
        }
      },
      "search": {
        "request": {
          "search_type": "query_then_fetch",
          "indices": [
            "indice_name*"
          ],
          "types": [
            "events"
          ],
          "body": {
            "size": 0,
            "_source": {
              "excludes": []
            },
            "aggs": {
              "uuid": {
                "terms": {
                  "field": "device_uuid",
                  "size": 10,
                  "order": {
                    "_term": "desc"
                  },
                  "min_doc_count": "10"
                },
                "aggs": {
                  "geo": {
                    "cardinality": {
                      "field": "best_geo"
                    }
                  },
                  "acc": {
                    "cardinality": {
                      "field": "best_accuracy"
                    }
                  },
                  "age": {
                    "cardinality": {
                      "script": {
                        "inline": "(doc['@timestamp'].date.getMillis() - doc['user_creation_time'].date.getMillis())/1000/60/60/24",
                        "lang": "painless"
                      }
                    }
                  }
                }
              }
            },
            "stored_fields": [
              "*"
            ],
            "script_fields": {
              "user_age": {
                "script": {
                  "inline": "(doc['@timestamp'].date.getMillis() - doc['user_creation_time'].date.getMillis())/1000/60/60/24",
                  "lang": "painless"
                }
              }
            },
            "docvalue_fields": [
              "@timestamp",
              "time",
              "user_creation_time"
            ],
            "query": {
              "bool": {
                "must": [
                  {
                    "match_all": {}
                  },
                  {
                    "match_phrase": {
                      "gc": {
                        "query": 1
                      }
                    }
                  },
                  {
                    "match_phrase": {
                      "solution": {
                        "query": "first"
                      }
                    }
                  },
                  {
                    "range": {
                      "@timestamp": {
                        "gte": "now-1d",
                        "lte": "now"
                      }
                    }
                  }
                ],
                "filter": [],
                "should": [],
                "must_not": []
              }
            }
          }
        }
      }
    },
    "condition": {
      "type": "script",
      "status": "success",
      "met": false
    },
    "actions": []
  },
  "messages": []
}

_Sergey · July 17, 2018, 10:40am

I check what numbers I have in age in buckets doc, then put one of the number in condition for age, so to simulate to see that condition is met, but it says that met: false...

Btw, I tried to play with allmatch vs anymatch --> I tried to use this one:

" source": "return ctx.payload.aggregations.uuid.buckets.stream().anyMatch(b -> b.age.value == 3) ",

and it is alerting, But in text of notification in slack I using:

"{{#ctx.payload.aggregations.uuid.buckets}}{{key}}: {{geo.value}} {{/ctx.payload.aggregations.uuid.buckets}}" - it shows all docs in bucket, not only doc where age =3 (may be it is correct and I used incorrectly it...)

_Sergey · July 19, 2018, 7:37am

@spinscale, Hey Alex, have any thoughts on this, please?
Much appreciate your input!

spinscale · July 19, 2018, 8:26am

Be advised that this is a community forum and there are no service level agreements on replies and how long it takes, if at all.

If you need more timely support, Elastic is offering support subscriptions for exactly this.

_Sergey · July 24, 2018, 1:05pm

Hello, Sorry for my impatience.

I want to share some info that I managed to do here:

I decided to use chain input here: in first I check only aggs for geo and use bucket_filter =1 to show only docs where geo=1. In the second input I check only aggs for acc with the same bucket_filter =1 to show only docs where acc=1. And in the second input the same procedure for aggs for age but with >=2.

But my question is: if it possible to take user_name from first output to check if this name is presented in second output and if yes to check this name if it is presented in third one - then notify me if name is presented in every output.

_Sergey · July 24, 2018, 1:12pm

Hey,

I just used this condition:

condition": {
    "script": {
      "source": "if (ctx.payload.first.aggregations.uuid.key == ctx.payload.second.aggregations.uuid.key == ctx.payload.third.aggregations.uuid.key) return true; return false",
      "lang": "painless"    }

Did I cover all things, so every user_name (key) from first are comparing with second output and with third output ?

spinscale · July 25, 2018, 7:55am

You cannot compare three arrays like that. This script will not work at all. You need to gather the all keys for each search in a list and then can compare them with each other.

_Sergey · July 25, 2018, 8:52am

Can you share info how to do your advice, please

But my compare script seems work, at least when I compare two outputs: where I know that names are intersected it alerts, and where not - it doesn't alert. (I generate all keys and then check it in excel - to double check).

UPDATE: oh. really don't work my script, just simulate when 1 name is presented in every output but watcher doesn't alert

Thanks

system · August 22, 2018, 8:52am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Comparing the last aggregated sub bucket count in the watcher condition Kibana elastic-stack-alerting	2	290	July 27, 2021
Watcher - Write painless script for looping aggregated buckets Elasticsearch elastic-stack-alerting , painless	3	1073	January 18, 2023
Watcher not work perfect Elasticsearch elastic-stack-monitoring , elastic-stack-alerting	4	336	November 2, 2020
Watcher condition setup Elasticsearch	1	282	January 4, 2022
Check some condition for every document in buckets Elasticsearch elastic-stack-alerting	4	454	February 7, 2019

How to check some condition through all docs in aggregation

Related topics