Elastalert module not triggering alerts


(Kaviarasan A) #1

Hi Team,

I have been trying to get an alert when my CPU load value(system.load.1 >0.8). I could see the rule is parsed without any errors but I am unable to get any alerts/hits. Configuration files attached for your reference.

Elasticsearch_Logs

Screenshot shows I do receive logs with system.load.1 >0.8(Timeframe 10:33:00 - 10:34:00 ).

Output :

python -m elastalert.elastalert --verbose --rule cpu_high.yaml --config config.yaml
elastalert/config.py:31: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
rule_schema = jsonschema.Draft4Validator(yaml.load(open(os.path.join(os.path.dirname(file), 'schema.yaml'))))
INFO:elastalert:Starting up
INFO:elastalert:Ran Metricbeat CPU Spike Rule from 2019-03-17 10:19 PDT to 2019-03-17 10:30 PDT: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59.947278 seconds
INFO:elastalert:Ran Metricbeat CPU Spike Rule from 2019-03-17 10:30 PDT to 2019-03-17 10:31 PDT: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59.965511 seconds
INFO:elastalert:Ran Metricbeat CPU Spike Rule from 2019-03-17 10:31 PDT to 2019-03-17 10:32 PDT: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59.965717 seconds
INFO:elastalert:Ran Metricbeat CPU Spike Rule from 2019-03-17 10:32 PDT to 2019-03-17 10:33 PDT: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59.962178 seconds
INFO:elastalert:Ran Metricbeat CPU Spike Rule from 2019-03-17 10:33 PDT to 2019-03-17 10:34 PDT: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59.952755 seconds
INFO:elastalert:Ran Metricbeat CPU Spike Rule from 2019-03-17 10:34 PDT to 2019-03-17 10:35 PDT: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59.964988 seconds

Configuration file,

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: /home/elastalert/tmp/elastalert/example_rules

# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  minutes: 1

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 15

# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: 192.168.241.128

# The Elasticsearch port
es_port: 9200

# The AWS region to use. Set this when using AWS-managed elasticsearch
#aws_region: us-east-1

# The AWS profile to use. Use this if you are using an aws-cli profile.
# See http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
# for details
#profile: test

# Optional URL prefix for Elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to Elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword

# Use SSL authentication with client certificates client_cert must be
# a pem file containing both cert and key for client
#verify_certs: True
#ca_certs: /path/to/cacert.pem
#client_cert: /path/to/client_cert.pem
#client_key: /path/to/client_key.key

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

# Custom logging configuration
# If you want to setup your own logging configuration to log into
# files as well or to Logstash and/or modify log levels, use
# the configuration below and adjust to your needs.
# Note: if you run ElastAlert with --verbose/--debug, the log level of
# the "elastalert" logger is changed to INFO, if not already INFO/DEBUG.
#logging:
#  version: 1
#  incremental: false
#  disable_existing_loggers: false
#  formatters:
#    logline:
#      format: '%(asctime)s %(levelname)+8s %(name)+20s %(message)s'
#
#    handlers:
#      console:
#        class: logging.StreamHandler
#        formatter: logline
#        level: DEBUG
#        stream: ext://sys.stderr
#
#      file:
#        class : logging.FileHandler
#        formatter: logline
#        level: DEBUG
#        filename: elastalert.log
#
#    loggers:
#      elastalert:
#        level: WARN
#        handlers: []
#        propagate: true
#
#      elasticsearch:
#        level: WARN
#        handlers: []
#        propagate: true
#
#      elasticsearch.trace:
#        level: WARN
#        handlers: []
#        propagate: true
#
#      '':  # root logger
#        level: WARN
#          handlers:
#            - console
#            - file
#        propagate: false

Rule file :

name: Metricbeat CPU Spike Rule
type: metric_aggregation

#es_host: localhost
#es_port: 9200

index: metricbeat-*

buffer_time:
  minutes: 1

metric_agg_key: system.load.1
metric_agg_type: avg
query_key: beat.hostname
doc_type: metricsets
  
bucket_interval:
  minutes: 1
  
sync_bucket_interval: true
#allow_buffer_time_overlap: true
#use_run_every_query_size: true

min_threshold: 0.1
max_threshold: 0.8

#filter:
#- term:
#    metricset.name: load

# (Required)
# The alert is use when a match is found
alert:
- "debug"

Please let me know what is the missing piece here which makes the alerting fails. Thanks in advance.


(David Pilato) #2

You might have a better chance getting an answer by asking on elastalert project?


(Kaviarasan A) #3

@dadoonet

Thanks ! I have created a ticket in elastalert forum.


(system) closed #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.