Creating Search Query Based on Conditional Logic: A & B & C & (D | (E & F))

Hi, thanks for your time.

I'm having a bit of trouble creating a search query based on conditional logic.

Via a search query, I am fetching anomalies from the .ml-anomalies-shared index. I have a variety of anomaly detection jobs set up - some that use the count function, and others that do not.

I want to pass my search query a list of job IDs, as well as a time range and an initial record score filter. Then, for jobs that use the count function, I only want to return results where the actual value is above 20. Otherwise, I want to return all anomalies.

The business need here is that for anomalies were the actual count is below 20, I want to ignore them. For other anomalies that are based on the mean function (for example), I want all anomalies return.

In pseudo-code, I am attempting something like this:

IF

job_id IN c('job_id_1', 'job_id_2', 'job_id_3') &
1570453375402 <= timestamp >= 1570366975399 &
initial_record_score >= 90 &
(function_description != 'count' | (function_description = 'count' & actual_count >= 20))

THEN
return anomalies

Below is the search query I've gotten so far. I think it is close, but not quite correct.

    POST /.ml-anomalies-shared/_search
{
  "size": 50,
  "query": {
    "constant_score": {
      "filter": {
        "bool": {
          "must": [
            {
              "terms": {
                "job_id": [
                  "job_id_1",
                  "job_id_2",
                  "job_id_3"
                ]
              }
            },
            {
              "range": {
                "timestamp": {
                  "from": 157037178391,
                  "to": 1570458567819
                }
              }
            },
            {
              "range": {
                "initial_record_score": {
                  "gte": 90
                }
              }
            }
          ],
          "should": [
            {
              "bool": {
                "must_not": [
                  {
                    "term": {
                      "function_description": "count"
                    }
                  }
                ],
                "must": [
                  {
                    "term": {
                      "function_description": "count"
                    }
                  },
                  {
                    "range": {
                      "actual": {
                        "gte": 20
                      }
                    }
                  }
                ]
              }
            }
          ],
          "minimum_should_match" : 1
        }
      }
    }
  }
}

Thank for your help and time!

Why not just make the ML job ignore this situation with a Custom Rule? See this blog for reference:

Hey Rich, thanks for the suggestion.

I was actually just able to get the search to work as needed. In the future, I may switch over to using a Custom Rule as you suggested.

For future reference for anyone else, here's a potential solution:

POST /.ml-anomalies-shared/_search
{
  "size": 50,
  "query": {
"constant_score": {
  "filter": {
    "bool": {
      "must": [
        {
          "terms": {
            "job_id": [
              "job_id_1",
              "job_id_2",
              "job_id_3"
            ]
          }
        },
        {
          "range": {
            "timestamp": {
              "from": "2019-09-18T23:00:00Z",
              "to": "2019-09-19T02:00:00Z"
            }
          }
        },
        {
          "range": {
            "initial_record_score": {
              "gte": 0
            }
          }
        }
      ],
      "should": [
        {
          "range": {
            "actual": {
              "gte": 20
            }
          }
        },
        {
          "term": {
            "function": {
              "value": "mean"
            }
          }
        }
      ],
      "minimum_should_match": 1
    }
  }
}
  }
}
1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.