Filter after selecting top 5 hits

Elastic Stack Kibana
1

Hi,

I'm using the alerting feature in Kibana and I want to check if the last 5 consecutive values of a field exceed a threshold x but if I use a filter in my elastic query, it gets applied before the top N aggregation.

Is there a way in which I can apply the filter after or check if the last consecutive values exceed a threshold using some other selector or method? I don't want to check this in the trigger condition in painless because that will return all the documents in the ctx and not just the ones which exceeded the threshold which I want to display in my alert message.

I've been stuck with this for a while and I have only seen blog posts saying sub aggregation is not possible on top N so any help or work around would be much appreciated.

This is my query :

{ 
    "size": 500, 
    "query": { 
        "bool": { 
            "filter": [ 
                { 
                    "match_all": {
                        "boost": 1 
                    } 
                }, 
                {
                    "match_phrase": { 
                        "client.id": { 
                            "query": "42", 
                            "slop": 0, 
                            "zero_terms_query": "NONE", 
                            "boost": 1 
                        } 
                    } 
                }, 
                { 
                    "range": { 
                        "@timestamp": { 
                            "from": "{{period_end}}||-10m", 
                            "to": "{{period_end}}", 
                            "include_lower": true, 
                            "include_upper": true, 
                            "format": "epoch_millis", 
                            "boost": 1 
                        } 
                    }
                } 
            ], 
            "adjust_pure_negative": true, 
            "boost": 1 
        } 

    }, 
    "aggs": {
        "2": {
          "terms": {
            "field": "component.name",
            "order": {
              "_key": "desc"
            },
            "size": 50
          },
          "aggs": {
            "3": {
              "terms": {
                "field": "client.name.keyword",
                "order": {
                  "_key": "desc"
                },
                "size": 5
              },
              "aggs": {
                "1": {
                  "top_hits": {
                    "docvalue_fields": [
                      {
                        "field": "component.stats.gc.oldgen.used.pct",
                        "format": "use_field_mapping"
                      }
                    ],
                    "_source": "component.stats.gc.oldgen.used.pct",
                    "size": 5,
                    "sort": [
                      {
                        "@timestamp": {
                          "order": "desc"
                        }
                      }
                    ]
                  }
                }
              }
            }
          }
        }
      }
    }       
}

Thanks

2

Hello Bhavya, let me check this with the team

1 Like
3

Hi @Bhavya_Bansal! Can you provide more details - are you trying to create a watch using the Watcher UI in Kibana? Thanks!

4

Hi Alison,

I'm using the alerting feature of opendistro.

Thanks,
Bhavya

5

@Bhavya_Bansal Ah, I see. OpenDistro is using a fork of Elastic products and therefore we do not have details on the issues specific to it. I would recommend opening an issue at the OpenDistro forum.

6

However, the query works similar to Watcher and my requirement is the same. It would be really helpful if you could suggest something.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.