Find the hosts whose hits are 5% more/less than the mean of hits of all hosts

[naveensrikanth]

Hi,

I am new to ElasticSearch watcher scripts. Can you help me with the below problem:

We get hits/transactions to 50 hosts/nodes. We would like to send an email notification with the host names whose hits are 5% more or less than the average hits across all hosts.

Thanks
NS

[spinscale (Alexander Reelsen)]

Hey,

before writing any watch the main question is, can you write the correct queries. This is actually not a watcher issue at all, but more a question of data exploration.

A query could be a avg aggregation in the transaction field

A second aggregation in that query could be the transactions grouped by host using a terms aggregation.

Then you could use the condition with a painless script to check if there are hosts that differ by about %5.

In general, I would be careful with averages due to it's outlier specific nature. Maybe percentiles and using a percentile agg makes sense in your case, but this is hard to tell from the outside.

[naveensrikanth]

I would like to monitor the hits every 1 min so I need a watcher.

Is this approach correct ?

1. I will create 2 inputs (chained).
2. First returns total no.of hits in 1 min --- ctx.payload.hits.total gives that.
3. Second I create buckets (keys are hostnames). then i get the document count in each bucket (values are document count)
4. create a painless script to check if there are any hosts that differ by 5%

And I got stuck at the buckets. How do I loop through the buckets and get the count of each bucket.

Here is my script:

{
    "trigger": {
      "schedule": {
        "interval": "1m"
      }
    },
    "input": {
    "chain": {
    "inputs": [
      {
      "all":{
          "search": {
            "request": {
              "search_type": "query_then_fetch",
              "indices": [
                "90183*"
              ],
              "types": [],
              "body": {
                "query": {
                  "bool": {
                    "must": [
                      {
                        "range": {
                          "@timestamp": {
                            "from": "now-1m",
                            "to": null,
                            "include_lower": true,
                            "include_upper": true,
                            "boost": 1
                          }
                        }
                      }
                    ],
                    "should": [
                      {
                        "range": {
                          "total_flow_time": {
                            "from": 1,
                            "to": null,
                            "include_lower": true,
                            "include_upper": true,
                            "boost": 2
                          }
                        }
                      }
                    ],
                    "disable_coord": false,
                    "adjust_pure_negative": true,
                    "minimum_should_match": "1",
                    "boost": 1
                  }
                },
                "ext": {}
              }
            }
          }
        }
      },
      {
      "per_host":{
          "search": {
            "request": {
              "search_type": "query_then_fetch",
              "indices": [
                "90183*"
              ],
              "types": [],
              "body": {
                "query": {
                  "bool": {
                    "must": [
                      {
                        "range": {
                          "@timestamp": {
                            "from": "now-1m",
                            "to": null,
                            "include_lower": true,
                            "include_upper": true,
                            "boost": 1
                          }
                        }
                      }
                    ],
                    "should": [
                      {
                        "range": {
                          "total_flow_time": {
                            "from": 1,
                            "to": null,
                            "include_lower": true,
                            "include_upper": true,
                            "boost": 2
                          }
                        }
                        }
                    ],
					"aggs": {
					"origin": {
					"terms": {
					"field": "host.keyword"
						}
					  }
					},
					"extract": [
					"aggregations.origin.buckets.key",
				        "aggregations.origin.buckets.sum_bucket"
					],
                    "disable_coord": false,
                    "adjust_pure_negative": true,
                    "minimum_should_match": "1",
                    "boost": 1
                  }
                },
                "ext": {}
              }
            }
          }
        }
    }
    ]  
    }
    },
    "condition": {
      "script": "return true"
     },
    "actions": {
      "send_email": {
        "email": {
          "profile": "standard",
          "attachments": {
            "attached_data.txt": {
              "data": {
                "format": "json"
              }
            }
          },
          "priority": "high",
          "to": [
            "xyz@gmail.com"
          ],
          "subject": " {{ctx.payload.all.hits.total}} ",
          "body": {
            "html": "{{ctx.payload.aggregations}} "
          }
        }
      }
  }
}

[spinscale (Alexander Reelsen)]

I do not see a script in that watch, just an always true condition.

Please take some time and check the examples repository for some more examples using painless in conditions.

In general, you can use painless to check lists using java lambdas

ctx.payload.aggregations.my_bucket.buckets.stream().anyMatch(b -> b.path_to_value_to_check < threshold*1.05 && b.path_to_value_to_check > threshold*0.95)

The threshold needs to be extracted from the payload as well.

Hope this helps as a getting started point!

--Alex