Watcher Alert on Agent Log Halt

[vishakh]

I would request a suggestion on Kibana Watcher script to set alert trigger when a specific server's filebeat agent stops/halts ingesting logs to index.

Brief: Trigger alarm when we don't receive logs from a specific filebeat agent (server) for more than 5mins

[wangqinghuan (Wang)]

Do you have any troubles in using Watcher. Kibana also provide a Web UI to configure Watcher.

[vishakh]

@wangqinghuan From the UI, i couldn't see an option to select-define the agent-hostname onto which I can configure alarm for

Example: I've a server "nginx-master" having a filebeat installed on it. So, I would need to create alarm specifically for this server/agent so that I can receive alerts if this servers doesn't ingest (or) if ELK doesn't receive any logs from this server for a period of >5mins

[wangqinghuan (Wang)]

okay, try this script:

{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "must": [
                {
                  "term": {
                    "host.keyword": "nginx-host"
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "gte": "{{ctx.trigger.scheduled_time}}||-5m",
                      "lte": "{{ctx.trigger.scheduled_time}}",
                      "format": "strict_date_optional_time||epoch_millis"
                    }
                  }
                }
              ]
            }
          }
        },
        "indices": [
          "filebeats*"
        ]
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "eq": 0
      }
    }
  },
  "actions": {
    "my-logging-action": {
      "logging": {
        "text": "There are no document in your index for last 5 minutes."
      }
    }
  }
}

[vishakh]

Thank You @wangqinghuan