Transform free bytes into GB in ES watch for hard disk utilisation

[sahil_sawhney]

In my scenario i want to simply transform free bytes of Hard disk into GB(wrt to .monitoring-es-6* index). Doing this operation in groovy was a bit easy. But wrt to v5.6 of ES i need to use java 8 to do so.
Applieng map transforms the value but it do not return me back the object on which i was working on.
my watcher json is something like this

{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": [
          ".monitoring-es-6-*"
        ],
        "body": {
          "size" : 0,
          "query": {
            "bool": {
              "filter": {
                "range": {
                  "timestamp": {
                    "gte": "now-2m",
                    "lte": "now"
                  }
                }
              }
            }
          },
          "aggs": {
            "minutes": {
              "date_histogram": {
                "field": "timestamp",
                "interval": "1m"
              },
              "aggs": {
                "nodes": {
                  "terms": {
                    "field": "source_node.ip",
                    "order": {
                      "hdd": "desc"
                    }
                  },
                  "aggs": {
                    "hdd": {
                      "avg": {
                        "field": "node_stats.fs.total.free_in_bytes"
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "script":  "if (ctx.payload.aggregations.minutes.buckets.size() == 0) return false; def latest = ctx.payload.aggregations.minutes.buckets[-1]; def node = latest.nodes.buckets[0]; return node.hdd.value <= 7000000000000L;"
  },
  "actions": {
    "send_email": { 
      "transform": {
        "script": "def latest = ctx.payload.aggregations.minutes.buckets[-1]; return latest.nodes.buckets.stream().filter(it -> it.hdd.value <= 7000000000000L).collect(Collectors.toList());"
      },
      "email": {
        "to": "sahil.sawhney@knoldus.in", 
        "subject": "Watcher Notification - LOW FREE HARD DISK ON SATURN ES",
        "body": "Nodes with LOW FREE HARD DISK (BELOW 100):\n\n{{#ctx.payload._value}}\"{{key}}\" - FREE HARD DISK is {{hdd.value}}%\n{{/ctx.payload._value}}"
      }
    }
  }
}

Can anyone help me with the logic to implement this transformation?

[spinscale (Alexander Reelsen)]

hey,

how about sharing your groovy script and we transform it into painless?

--Alex

[sahil_sawhney]

hey @spinscale
following is my groovy script wrt es 2.4

def latest = ctx.payload.aggregations.minutes.buckets[-1]; return latest.nodes.buckets.findAll ({ return it.available_disk && it.available_disk.value >= 0 }).findAll({it.available_disk.percent=Math.round((it.available_disk.value/it.total_disk.value)*100); it.available_disk.percent < 15 })

i want to transform for it.available_disk.percent

[spinscale (Alexander Reelsen)]

hey.

ok, so you have to convert your groovy closures into java lambdas (which painless reuses).

Something like this to give you a first hint

def last = ctx.payload.aggregations.foo.buckets.get(ctx.payload.aggregations.foo.buckets.size()-1);
def hosts = last.nodes.buckets.stream()
  .filter(b -> b.available_disk != null && b.available_disk.value > 0)
  .filter(b -> Math.round(b.available_disk.value/b.total_disk.value)*100 < 100)
  .map(b -> b.key)
  .collect(Collectors.toList());
return [ 'hosts': hosts]

[sahil_sawhney]

Hey @spinscale
i have made certain amendments to the snippet you sent

def latest = ctx.payload.aggregations.minutes.buckets.get(ctx.payload.aggregations.minutes.buckets.size() - 1);
def result = latest.nodes.buckets.stream().filter(it -> it != null && it.free_disk != null && it.free_disk.value > 0).filter(it -> {float free_percent = (float) it.free_disk.value / it.total_disk.value * 100;  return free_percent < 100;}).map(it -> it.key).collect(Collectors.toList());
return result; 

Now the problem is that the map statement transforms the payload and leave only the ip address of the host in the payload.
But i want to

1. Introduce new filed in payload, keeping the original ones(want to introduce the field free_percent)
2. transform the field available_in_bytes from byte count to GB count.

[spinscale (Alexander Reelsen)]

just return the original payload and add a field.

"script" "def payload = ctx.payload; payload.hosts = ... ; return payload"

[sahil_sawhney]

@spinscale Got a way to transform the payload fields. We have to do something like

def latest = ctx.payload.aggregations.minutes.buckets.get(ctx.payload.aggregations.minutes.buckets.size() - 1);
def result = latest.nodes.buckets.stream().filter(it -> it != null && it.free_disk != null && it.free_disk.value > 0).filter(it -> {it.free_disk.value = it.free_disk.value / 1000000000;it.total_disk.value = it.total_disk.value / 1000000000; return (float) it.free_disk.value / it.total_disk.value * 100 < 100;}).collect(Collectors.toList());
return result;

[sahil_sawhney]

So below is an example which shows how field can be transformed as well as added into payload

def latest = ctx.payload.aggregations.minutes.buckets.get(ctx.payload.aggregations.minutes.buckets.size() - 1);
def result = latest.nodes.buckets.stream().filter(it -> it != null && it.free_disk != null && it.free_disk.value > 0).filter(it -> {def res = (float) it.free_disk.value / it.total_disk.value * 100; it.free_disk.percent = Math.round(res * 100.0) / 100.0; it.free_disk.value = Math.round((it.free_disk.value / 1000000000) * 100.0) / 100.0; it.total_disk.value = Math.round((it.total_disk.value / 1000000000) * 100.0) / 100.0; return res < 100;}).collect(Collectors.toList());
return result;

By this i would conclude that painless is something which merges functionality provided by java 8 and groovy(as per 2.4).