Watcher, X amount of logs, per X host per hour

Elastic Stack Elasticsearch
1

Hello, I have been fighting with this for a few hours now and just can't get an alert that works.

I want to basically find out when a host is spamming syslog, so I have a watcher like so:

{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "filter": {
                "range": {
                  "@timestamp": {
                    "gte": "{{ctx.trigger.scheduled_time}}||-1h",
                    "lte": "{{ctx.trigger.scheduled_time}}",
                    "format": "strict_date_optional_time||epoch_millis"
                  }
                }
              }
            }
          },
          "aggs": {
            "bucketAgg": {
              "terms": {
                "field": "host.name",
                "size": "1",
                "order": {
                  "_count": "desc"
                }
              }
            }
          }
        },
        "indices": [
          "system-logs-*"
        ]
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gte": 500
      }
    }
  }
}

However when I run this, it fails to perform any actions with the error:

"error": {
          "root_cause": [
            {
              "type": "general_script_exception",
              "reason": "Error running com.github.mustachejava.codes.DefaultMustache@8a1d0f5"
            }
          ],
          "type": "general_script_exception",
          "reason": "Error running com.github.mustachejava.codes.DefaultMustache@8a1d0f5",
          "caused_by": {
            "type": "mustache_exception",
            "reason": "Failed to get value for ctx.payload.hits.hits.0._source.host.name @[query-template:1]",
            "caused_by": {
              "type": "mustache_exception",
              "reason": "0 @[query-template:1]",
              "caused_by": {
                "type": "index_out_of_bounds_exception",
                "reason": "0"
              }
            }
          }

Am I missing something blindly obvious here or am I just not understanding how to correct persue it? I am migrating from elastalert to watcher, this is how my elastalert looked like:

# Check a single host for over 500 syslog messages in an hour
name: Spamming Syslog Messages (1 Hour) - Warning
filter:
  - term:
      source: /var/log/syslog
type: frequency
num_events: 500
timeframe:
  hours: 1
query_key: beat.hostname

# (Required)
# Index to search, wildcard supported
index: system-logs-*

terms_window_size:
  days: 1
2

I feel like my information is somewhat lacking so I'll just cover what my alert should do.

Check index for over 500 items that belong to:

system-logs-*

Field: [source: /var/log/syslog]

Number of documents >= 500

per host.

So if a host has over 500 entries in elasticsearch with the same source record, then it should alert.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.