Multiple queries - action on second query if not met after x timeframe

Forgive me if this is already answered somewhere else on this forum, but I haven't had much luck finding it.

I'm trying to create a watcher to check for events (ie: down node). From within this watcher, i ONLY want to make an event if I haven't received a second event with a specific response within x time.

ex:
look messages related to node "test" - search for the "disconnect" message. if I don't receive a "cleared" message for this same node within 15 minutes of receiving the disconnect - do an action.

my start point thus far:

{
  "trigger": {
    "schedule": {
      "interval": "30m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 10000,
          "query": {
            "bool": {
              "must": [
                {
                  "match_phrase": {
                    "message.alarmType": {
                      "query": "disconnect"
                    }
                  }
                },
                {
                  "match_phrase": {
                    "message.alarmKey": {
                      "query": "DEVICE_NAME"
                    }
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "format": "strict_date_optional_time",
                      "gte": "now-30m"
                    }
                  }
                }
              ],
              "filter": [
                {
                  "match_all": {}
                }
              ],
              "should": [],
              "must_not": [
                {
                  "match_phrase": {
                    "message.alarmClass": {
                      "query": "cleared"
                    }
                  }
                }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gte": 10
      }
    }
  },
  "actions": {
    "my-logging-action": {
      "logging": {
        "level": "info",
        "text": "There are {{ctx.payload.hits.total}} documents in your index. Threshold is 10."
      }
    }
  }
}

you can use the chain input to execute multiple queries as an input for a watch.

Another idea would be to execute a query with aggregations. One aggregation would aggregate on DOWN events and hostnames and another on UP events an hostnames. This way you could check which hostnames are only in the DOWN section and thus knew, that there was no up event.

Would that make sense?

--Alex

Thanks for the ideas Alex, I'll give them a shot!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.