Hi Team,
I have been trying to get an alert when my CPU load value(system.load.1 >0.8). I could see the rule is parsed without any errors but I am unable to get any alerts/hits. Configuration files attached for your reference.
Screenshot shows I do receive logs with system.load.1 >0.8(Timeframe 10:33:00 - 10:34:00 ).
Output :
python -m elastalert.elastalert --verbose --rule cpu_high.yaml --config config.yaml
elastalert/config.py:31: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
rule_schema = jsonschema.Draft4Validator(yaml.load(open(os.path.join(os.path.dirname(file), 'schema.yaml'))))
INFO:elastalert:Starting up
INFO:elastalert:Ran Metricbeat CPU Spike Rule from 2019-03-17 10:19 PDT to 2019-03-17 10:30 PDT: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59.947278 seconds
INFO:elastalert:Ran Metricbeat CPU Spike Rule from 2019-03-17 10:30 PDT to 2019-03-17 10:31 PDT: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59.965511 seconds
INFO:elastalert:Ran Metricbeat CPU Spike Rule from 2019-03-17 10:31 PDT to 2019-03-17 10:32 PDT: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59.965717 seconds
INFO:elastalert:Ran Metricbeat CPU Spike Rule from 2019-03-17 10:32 PDT to 2019-03-17 10:33 PDT: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59.962178 seconds
INFO:elastalert:Ran Metricbeat CPU Spike Rule from 2019-03-17 10:33 PDT to 2019-03-17 10:34 PDT: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59.952755 seconds
INFO:elastalert:Ran Metricbeat CPU Spike Rule from 2019-03-17 10:34 PDT to 2019-03-17 10:35 PDT: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59.964988 seconds
Configuration file,
# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: /home/elastalert/tmp/elastalert/example_rules
# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
minutes: 1
# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
minutes: 15
# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: 192.168.241.128
# The Elasticsearch port
es_port: 9200
# The AWS region to use. Set this when using AWS-managed elasticsearch
#aws_region: us-east-1
# The AWS profile to use. Use this if you are using an aws-cli profile.
# See http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
# for details
#profile: test
# Optional URL prefix for Elasticsearch
#es_url_prefix: elasticsearch
# Connect with TLS to Elasticsearch
#use_ssl: True
# Verify TLS certificates
#verify_certs: True
# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET
# Option basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword
# Use SSL authentication with client certificates client_cert must be
# a pem file containing both cert and key for client
#verify_certs: True
#ca_certs: /path/to/cacert.pem
#client_cert: /path/to/client_cert.pem
#client_key: /path/to/client_key.key
# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status
# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
days: 2
# Custom logging configuration
# If you want to setup your own logging configuration to log into
# files as well or to Logstash and/or modify log levels, use
# the configuration below and adjust to your needs.
# Note: if you run ElastAlert with --verbose/--debug, the log level of
# the "elastalert" logger is changed to INFO, if not already INFO/DEBUG.
#logging:
# version: 1
# incremental: false
# disable_existing_loggers: false
# formatters:
# logline:
# format: '%(asctime)s %(levelname)+8s %(name)+20s %(message)s'
#
# handlers:
# console:
# class: logging.StreamHandler
# formatter: logline
# level: DEBUG
# stream: ext://sys.stderr
#
# file:
# class : logging.FileHandler
# formatter: logline
# level: DEBUG
# filename: elastalert.log
#
# loggers:
# elastalert:
# level: WARN
# handlers: []
# propagate: true
#
# elasticsearch:
# level: WARN
# handlers: []
# propagate: true
#
# elasticsearch.trace:
# level: WARN
# handlers: []
# propagate: true
#
# '': # root logger
# level: WARN
# handlers:
# - console
# - file
# propagate: false
Rule file :
name: Metricbeat CPU Spike Rule
type: metric_aggregation
#es_host: localhost
#es_port: 9200
index: metricbeat-*
buffer_time:
minutes: 1
metric_agg_key: system.load.1
metric_agg_type: avg
query_key: beat.hostname
doc_type: metricsets
bucket_interval:
minutes: 1
sync_bucket_interval: true
#allow_buffer_time_overlap: true
#use_run_every_query_size: true
min_threshold: 0.1
max_threshold: 0.8
#filter:
#- term:
# metricset.name: load
# (Required)
# The alert is use when a match is found
alert:
- "debug"
Please let me know what is the missing piece here which makes the alerting fails. Thanks in advance.