Add a condition on ML

Hello ,
I m using ML plugin Kibana 7.4.

I have a question about the following Case:

  • We have a list of Event.
  • We are reassuring the number of occurrence of these event.
  • All the event are important
  • But the events are not important if there is specific event that occurs on the middle of a list of events:
    For example :
  • the number of event is important if i have the following sequence : X X X X Y Y X X X X Y X X
  • the number of event is not important if i have the following sequence : X X X X Y Y X X X X Y X Z X
    Where X Y and Z are my events name.
    The second sequence is not important since that i have and event named Z received in the sequence.

Is it possible to change the score of the ML on this case ? so that we avoid alerting on this case.

Thanks
Best regards
Amine S

At what frequency do these events occur? Do they all happen within the same minute? hour? day? Or an unknown, arbitrary time? This will be important, I think because to assess whether or not some kind of event "is in the middle of others", then you'll need to wait until the subsequent events appear or not. How long you need to wait to determine this situation will be important. At this point, I'm not even entirely convinced this is an ML problem - perhaps it can just be solved with a search and a conditional.

Thanks for your prompt response

1-This event occur when the operator set something manually and often

2- this event happen before 1 to 2 min before the other events occur and I want to ignore all them

3- We have configured a bucket span of 5 minutes

Thanks

Hi
I'm also facing the same problem " how to add condition on ML?"
Thanks

@AmS - I'm still unconvinced this is a use-case best solved by ML. It seems to me that you can solve this use case by using the sequence search of EQL. (EQL is available v7.9+)

Something like:

GET /events/_eql/search
{
  "query": """
    sequence
      [ myevent where event.value == "X" ]
      [ myevent where event.value == "Y"  ]
      [ myevent where event.value == "Z" ]
  """
}

If the events were

X X X X Y Y X X X X Y X X - the above query would return nothing

If the events were

X X X X Y Y X X X X Y X Z X - the above query would return the sequence

Then, within an alert (Watch) you could inspect the output of the EQL query and alert accordingly

Here's a simplified example - use DevTools to execute each command in order:

#create a index of events with necessary mappings
PUT events/
{
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date",
        "format": "yyyy-MM-dd HH:mm:ss"
      },
      "event": {
        "properties": {
          "category": {
            "type": "keyword"
          },
          "value": {
            "type": "keyword"
          }
        }
      }
    }
  }
}

PUT events/_doc/1
{
  "@timestamp": "2020-11-23 12:00:00",
  "event.category": "myevent",
  "event.value": "X"
}
PUT events/_doc/2
{
  "@timestamp": "2020-11-23 12:01:00",
  "event.category": "myevent",
  "event.value": "X"
}
PUT events/_doc/3
{
  "@timestamp": "2020-11-23 12:02:00",
  "event.category": "myevent",
  "event.value": "X"
}
PUT events/_doc/4
{
  "@timestamp": "2020-11-23 12:03:00",
  "event.category": "myevent",
  "event.value": "Y"
}
PUT events/_doc/5
{
  "@timestamp": "2020-11-23 12:04:00",
  "event.category": "myevent",
  "event.value": "X"
}
#Try searching the sequence - should see no results
GET /events/_eql/search
{
  "query": """
    sequence
      [ myevent where event.value == "X" ]
      [ myevent where event.value == "Y"  ]
      [ myevent where event.value == "Z" ]
  """
}

#Add in the critical event "Z"
PUT events/_doc/6
{
  "@timestamp": "2020-11-23 12:05:00",
  "event.category": "myevent",
  "event.value": "Z"
}
#Try search the sequence again - should now see the sequence returned
GET /events/_eql/search
{
  "query": """
    sequence
      [ myevent where event.value == "X" ]
      [ myevent where event.value == "Y"  ]
      [ myevent where event.value == "Z" ]
  """
}