Agrregation on multiple input in elasticsearch

Elastic Stack Elasticsearch
1

Elasticsearch version: 2.4.1

Plugins installed: watcher

Description of the problem:

I am monitoring the input sources and trying to output the sources which are active.

I have 3 input sources ["192.168.48.8", "192.168.30.150", "192.168.190.8"], out of which "192.168.190.8" is inactive,

I was able to get the name of active sources in my alert.

Watch Output:

"Input Sources: {0=192.168.48.8, 1=192.168.30.150, 2=192.168.190.8}

Some sources are not contributing;

Active Sources: {0={doc_count=52729, key=192.168.30.150}, 1={doc_count=40149, key=192.168.48.8}}"

Doubt:

Is there a way to notify the names of inactive sources (may be by subtracting the aggregated active sources from Input sources ), below is the watch implemented.

Thanks in advance!!!

Watch:

"trigger"
  {
    "schedule" : { "interval" : "1m" }
  },
  "input" : {
    "chain": {
      "inputs": [
        {
          "first": {
            "simple" : {
              "Input_hosts" : ["192.168.48.8", "192.168.30.150", "192.168.190.8"]
            }
          }
        },
        {
          "second": {
            "search" : {
              "request" : {
                "indices" : [ "firewalls" ],
                "body" : {
                  "query" : {
                    "bool": {
                      "must": [
                {
                          "range": {
                            "@timestamp": {
                              "gte": "now-4d"
                            }
                          }
                        },
                { "bool": {
                        "should": [
                              { "bool": { "must_not": { "match" : {"host": "192.168.190.8"}}}}, 
                              { "bool": { "must_not": {"match" : {"host": "192.168.48.8"}}}},
                              { "bool": { "must_not": {"match" : {"host": "192.168.30.150"}}}}]
                          }           
                        }
                      ]
                    }   
                  },
                  "size": 0,
                  "aggregations": {
                            'hosts': { "terms": { "field": "host" } }
                  }     
                }
              }
            }
          }
        }
      ]
    } 
  },
  "condition" : {
    "compare" : { "ctx.payload.second.hits.total" : { "gt" : 0 }} 
  },
  "throttle_period" : "12h",
  "actions" : {
    "send_email" : {     
      "email" : {
        "to" : ["abc@def.com"],
        "subject" : "{{ctx.watch_id}}",
        "body" : "Input Sources: {{ctx.payload.first.Input_hosts}}\n\nSome sources are not contributing;\n\nActive Sources: {{ctx.payload.second.aggregations.hosts.buckets}}",
        "attach_data" : true
      }
    }
  }
}'
2

Hey,

you could use a script transform to change the payload using a script and do that mentioned substraction.

See the transform script documentation for more.

--Alex

3

Can you please suggest me how to script it

Thanks,
Sravan

4

Hey.

if you are on groovy, you can just use the - operator to substract lists. See this snippet

 a = [ 1, 2, 3]
b = [ [key:"foo", value: 1], [key:"bar", value:2], ["key":"baz", value: 4]]

a - b*.value
===> [3]

--Alex