Generate Alert for entries not present in the log for a given time window

[akhettar (Ayache Khettar)]

Hello there,

I am hoping if somebody can help me write monitor's query to generate an alert to be pushed to a slack channel. I am using ELK stack v 7.4.2.

Here is the payload of the message:

{
   "metaData":{
      "timestamp":"2020-08-21T08:04:24.523Z",
      "appVersion":"0.0.1-SNAPSHOT",
      "appName":"cr-inventory-files-uploader",
      "logger":"http-nio-8080-exec-5",
      "priority":"INFO",
      "envName":"dev",
      "envHost":"localhost",
      "tracePoint":"START"
   },
   "payload":{
      "class":"com.vfc.mkpl.zacr.inventoryfilesuploader.controllers.FileUploadController:47",
      "message":"Received new inventory file: INVENTORY-PRICE_S307_20200716183709.csv",
      "customData":{

      },
      "exception":""
   },
   "context":{
      "correlationRootId":"edbec86e-210f-4529-abd0-e1cdc28266e6",
      "customData":{
         "file-size":74,
         "file-name":"INVENTORY-PRICE_S307_20200716183709.csv",
         "store-id":"S307",
         "user-agent":"AmazonAPIGateway_3tlidsdlq9"
      },
      "correlationId":"23a2a574-1151-4214-85da-f680664c6540"
   }
}

Basically the Monitor's query needs to check the presence of the above log for a given time window (1minute) and if not present it raises an alert. See below sample query I wrote but it is matching in all cases even if the log entry is present. The must clause works but must_not is not working. Have I missed anything?

thanks in advance

{
    "size": 0,
    "query": {
        "bool": {
            "must_not": [
               
                {
                    "match_phrase": {
                        "customData.store-id": {
                            "query": "S307",
                            "slop": 0,
                            "zero_terms_query": "NONE",
                            "boost": 1
                        }
                    }
                },
                {
                    "match_phrase": {
                        "metaData.tracePoint": {
                            "query": "START",
                            "slop": 0,
                            "zero_terms_query": "NONE",
                            "boost": 1
                        }
                    }
                },
                {
                    "range": {
                        "@timestamp": {
                            "from": "now-1m",
                            "to": "now",
                            "include_lower": true,
                            "include_upper": true,
                            "format": "strict_date_optional_time",
                            "boost": 1
                        }
                    }
                }
            ],
            "adjust_pure_negative": true,
            "boost": 1
        }
    }
}

[wylie (Wylie Conlon)]

You probably need to set minimum_should_match to 3, if you want all clauses to match. https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html

[akhettar (Ayache Khettar)]

@wylie

minimum_should_match

Many thanks Wylie, I tried that but it is not working. Alerts are still being triggered despite the fact there are entries in the logs

[wylie (Wylie Conlon)]

Reading your query again, it seems like you are excluding all documents from the last 1 minute, but your query will still match older documents. I think you want a query like this:

{
  "size": 0,
  "query": {
    "bool": {
      "must": [{
        "range": {
          "@timestamp": {
            "from": "now-1m",
            "to": "now",
            "include_lower": true,
            "include_upper": true,
            "format": "strict_date_optional_time",
            "boost": 1
          }
        }
      }],
      "must_not": [
        {
          "match_phrase": {
            "customData.store-id": {
              "query": "S307",
              "slop": 0,
              "zero_terms_query": "NONE",
              "boost": 1
            }
          }
        },
        {
          "match_phrase": {
            "metaData.tracePoint": {
              "query": "START",
              "slop": 0,
              "zero_terms_query": "NONE",
              "boost": 1
            }
          }
        },
      ],
      "minimum_should_match": 3,
      "adjust_pure_negative": true,
      "boost": 1
    }
  }
}

[akhettar (Ayache Khettar)]

Thanks again Wylie for your suggestion. I did try this but it is not matching, I do't know really what I am missing. I am thinking of rewritting the query to look for something in the log rather check if it's not there. I don't have such thing in the log right now, it has to be generated by the application hence I started with must-not query.

[wylie (Wylie Conlon)]

I agree that it would be easier to create positive matches, since you could easily see when it's successful.