Kibana watcher sending alert cpu utilisation more than 90%

Hi
I have configured Kibana watcher to monitor cpu utilisation and its sending alert to out monitoring team stating cpu utilisation ec2 instance(Worker node) more than 90% but when I have checked in AWS CloudWatch cpu utilisation not showing more than 50%.
We are using filebeat, metricbeat and heartbeat and I am using Elasticsearch v 7.16.3

Can you please tell me
Is there any discrepancy in cpu utilisation in Kibana and CloudWatch ?

Welcome to our community! :smiley:

Can you share a few images of the stack Monitoring graphs and the cloudwatch ones showing the discrepency?

Thanks for your qucik response warkolam

We are using kibana watcher to monitor cpu and please find kibana watcher and AWS CloudWatch screen shot


CPU Monitoring watcher jason file

{
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "metricbeat-*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "filter": {
                "range": {
                  "@timestamp": {
                    "gte": "{{ctx.trigger.scheduled_time}}||-5m",
                    "lte": "{{ctx.trigger.scheduled_time}}",
                    "format": "strict_date_optional_time||epoch_millis"
                  }
                }
              }
            }
          },
          "aggs": {
            "bucketAgg": {
              "terms": {
                "field": "kubernetes.node.name",
                "size": "40",
                "order": {
                  "metricAgg": "asc"
                }
              },
              "aggs": {
                "metricAgg": {
                  "max": {
                    "field": "kubernetes.node.cpu.usage.nanocores"
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "script": {
      "source": "ArrayList arr = xx.xx.aggregations.bucketAgg.buckets; for (int i = 0; i < arr.length; i++) { if (arr[i]['metricAgg'].value >= params.threshold) { return true; } } return false;",
      "lang": "painless",
      "params": {
        "threshold": 7200000000
      }
    }
  },
  "actions": {
    "alert_webhook": {
      "webhook": {
        "scheme": "http",
        "host": "xxxxxxxx.monitoring-xxxx-alert-proxy.svc.cluster.local",
        "port": 80,
        "method": "post",
        "path": "/api/alert",
        "params": {},
        "headers": {},
        "body": "\"Watcher id: {{ctx.watch_id}}\n\nWatcher entity_id: {{ctx.id}}\n\nTriggered_time: {{ctx.trigger.triggered_time}}\n\nEvent: {{ctx.watch_id}}: alert for error: \n\n{{ctx.payload.results}}"
      }
    }
  },
  "throttle_period_in_millis": 28800000
}