Exclusion List for Watcher

Elastic Stack Elasticsearch
1

Hi,

I am trying to make an index which could be used as a static list of fields with data that will be used as exclusions in watcher alerts.

At the moment, I am doing that using exclude in the watcher. But for a larger list is it possible to be done using some sort of lookup. That can be used as a whitelist or blacklist in the watcher.

Also, will the use lookup list significant impact on performance. What would be the best way of doing it if any.

Thanks in advance!

Ankit

2

How about the Terms Lookup Query? Would that help you?

3

Hi Alex,

This is useful, but putting it in a watch condition to compare against matching terms from search in the watch is something I am trying to figure out.

Maybe I need to look at it in a different way. Probably matching arrays in the watch search against the lookup index. Any leads if you could share to compare specific keys of one doc to another index's.

Thanks,
Ankit

4

You could also use a chain input, then execute a search to look up those filter values, and use that response in the second search for exclusion.

1 Like
5

Hi Alex,

I am trying to implement a List lookup functionality using the same approach, but somehow painless is not working for me. The triple quotes as mentioned in documentation for condition context doesn't work on watcher.

{
"trigger": {
  "schedule": {
    "interval": "1m"
    }
  },

"input": {
    "chain": {
      "inputs": [ 
        {
          "first": {    
            "search": {
            "request": {
              "search_type": "query_then_fetch",
              "indices": ["packetbeat-*"],
              "body": {
                "query": {
                      "bool": {
                        "filter": [
                            
                            { "range": {
                            "@timestamp": {"gte": "now-5m"}
                                }
                            }
                    ]
                  }  
                },
                "size": 0,
                "aggs": {
                    "source_ip":{
                        "terms" :{
                            "field": "source.ip"
                        }
                    }
                }

              }
            }
            }
          }  
        },      
        {
          "second": {
            "search": {
            "request": {
              "search_type": "query_then_fetch",
              "indices": ["ioc_list"],
              "body": {
                "query": {
                  "match_all": {}
                },
                "size": 0,
                "aggs": {
                    "ioc_ip":{
                        "terms" :{
                            "field": "ip"
                        }
                    }
                }
              }

            }
            }
          }
        }    
      ]
    }     
},

"condition": {
    "script": 
          """
              int matches = 0;
              int var = 0;
              for (int i=0; i < ctx.payload.first.aggregations.source_ip.buckets.length; i++) {
                  for ( int j=0; i < ctx.payload.first.aggregations.ioc_ip.buckets.length; j++ ) {
                    if (ctx.payload.first.aggregations.source_ip.buckets.[i].key == ctx.payload.second.aggregations.ioc_ip.buckets.[j].key) {
                      matches++ ;
                      var = i;
                    }
                }      
              }        
              if ( matches >= 1) {
                  return ctx.payload.first.aggregations.source_ip.buckets.[var].key;
              }       
          """
        
       
}
6

Just to give this a closure for anyone who goes down this path:

Issue 1 : Version 7.5.2 and below does not support triple quotes in script in the Kibana Watcher UI condition context, it's a reported bug on Github and probably would be fixed in 7.7. Workaround was to use watcher API to test and write the watch

Issue 2: The script had typos and return type was incorrect. Updated working script is below:

"condition": {
    "script": 
          """
              int matches = 0;
              for (int i=0; i < ctx.payload.first.aggregations.source_ip.buckets.size(); i++) {
                  for ( int j=0; j < ctx.payload.second.aggregations.ioc_ip.buckets.size(); j++ ) {
                    if (ctx.payload.first.aggregations.source_ip.buckets[i].key == ctx.payload.second.aggregations.ioc_ip.buckets[j].key) {
                      matches += 1;
                    }
                }      
              }        
              if ( matches >= 1) {
                  return true;
              }       
          """
        
       
}

Glad was able to find a solution after going around for a month (not full time ofcourse) for a feature which is very crucial in security analytics !
Code could be better I agree, I am not a coder by profession but a security practitioner. Feel free to give suggestions.

Cheers!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.