Kibana watcher sending alert cpu utilisation more than 90%

[sanjaychahar (Sanjay)]

Hi
I have configured Kibana watcher to monitor cpu utilisation and its sending alert to out monitoring team stating cpu utilisation ec2 instance(Worker node) more than 90% but when I have checked in AWS CloudWatch cpu utilisation not showing more than 50%.
We are using filebeat, metricbeat and heartbeat and I am using Elasticsearch v 7.16.3

Can you please tell me
Is there any discrepancy in cpu utilisation in Kibana and CloudWatch ?

[warkolm (Mark Walkom)]

Welcome to our community!

Can you share a few images of the stack Monitoring graphs and the cloudwatch ones showing the discrepency?

[sanjaychahar (Sanjay)]

Thanks for your qucik response warkolam

We are using kibana watcher to monitor cpu and please find kibana watcher and AWS CloudWatch screen shot

CPU Monitoring watcher jason file

{
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "metricbeat-*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "filter": {
                "range": {
                  "@timestamp": {
                    "gte": "{{ctx.trigger.scheduled_time}}||-5m",
                    "lte": "{{ctx.trigger.scheduled_time}}",
                    "format": "strict_date_optional_time||epoch_millis"
                  }
                }
              }
            }
          },
          "aggs": {
            "bucketAgg": {
              "terms": {
                "field": "kubernetes.node.name",
                "size": "40",
                "order": {
                  "metricAgg": "asc"
                }
              },
              "aggs": {
                "metricAgg": {
                  "max": {
                    "field": "kubernetes.node.cpu.usage.nanocores"
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "script": {
      "source": "ArrayList arr = xx.xx.aggregations.bucketAgg.buckets; for (int i = 0; i < arr.length; i++) { if (arr[i]['metricAgg'].value >= params.threshold) { return true; } } return false;",
      "lang": "painless",
      "params": {
        "threshold": 7200000000
      }
    }
  },
  "actions": {
    "alert_webhook": {
      "webhook": {
        "scheme": "http",
        "host": "xxxxxxxx.monitoring-xxxx-alert-proxy.svc.cluster.local",
        "port": 80,
        "method": "post",
        "path": "/api/alert",
        "params": {},
        "headers": {},
        "body": "\"Watcher id: {{ctx.watch_id}}\n\nWatcher entity_id: {{ctx.id}}\n\nTriggered_time: {{ctx.trigger.triggered_time}}\n\nEvent: {{ctx.watch_id}}: alert for error: \n\n{{ctx.payload.results}}"
      }
    }
  },
  "throttle_period_in_millis": 28800000
}