Help converting Watcher Alert to Painless

Elastic Stack Elasticsearch
1

I'm having real trouble working out how to convert my Watcher alerts to something that will run in xpack, specifically around converting to Painless. For example, I have an alert that should put a message on Slack if the memory of anode has been over 75% for 1 hour:

PUT _xpack/watcher/watch/cluster_memory_alert
{
"trigger": {
"schedule": {
"interval": "30m"
}
},
"input": {
"search": {
"request": {
"indices": [
".monitoring-es-*"
],
"types" : [
"node_stats"
],
"body": {
"size" : 1,
"query": {
"filter": {
"range": {
"timestamp": {
"gte": "now-2h",
"lte": "now"
}
}
}
},
"aggs": {
"minutes": {
"date_histogram": {
"field": "timestamp",
"interval": "10m"
},
"aggs": {
"nodes": {
"terms": {
"field": "source_node.name",
"size": 10,
"order": {
"memory": "desc"
}
},
"aggs": {
"memory": {
"max": {
"field": "node_stats.jvm.mem.heap_used_percent"
}
}
}
}
}
}
}
}
}
}
},
"throttle_period": "2h",
"condition": {
"script": {
"id": "clusterMemoryCondition"
}
},
"actions": {
"notify-slack": {
"transform": {
"script": {
"id": "clusterMemoryTransform"
}
},
"slack": {
"message" : {
"to": [ "#monitoring" ],
"text": "Elastic Cloud Nodes with HIGH MEMORY Usage (above 75%) for the last hour:\n\n{{#ctx.payload._value}}"{{key}}" - Memory Usage is at {{memory.value}}%\n{{/ctx.payload._value}}"
}
}
}
}
}

The condition and transform scripts are:

POST /_scripts/groovy/clusterMemoryCondition
{
"script": "if (ctx.payload.hits.total < 1) return false; def rows = ctx.payload.hits.hits; if (rows[0]._source.node_stats.jvm.mem.heap_used_percent < 75) return false; if (ctx.payload.aggregations.minutes.buckets.size() < 12) return false; def last60Minutes = ctx.payload.aggregations.minutes.buckets[-12..-1]; return last60Minutes.every { it.nodes.buckets.every { s -> s.memory.value >= 75 }}"
}

And

POST /_scripts/groovy/clusterMemoryTransform
{
"script": "def latest = ctx.payload.aggregations.minutes.buckets[-1]; return latest.nodes.buckets.findAll { return it.memory && it.memory.value >= 75 };"
}

Any assistance in how to get those two scripts to work in Painless would be much appreciated

2

Heya,

we are currently in the process of revamping our painless documentation to make it as awesome as the scripting language itself. Right now, it is really hard, which language/java features are supported and which are not. You can check out this directory in the source to get a list of supported methods, but it is just a start of course.

Have you tried anything yet, that we can try to correct or are you starting at zero?

--Alex

3

At the moment I don't really know what I'm doing - I'm just parroting alerts and scripts from the documentation in the hopes they'll work. In the example above, what I'm aiming for is an alert that fires if the memory on any node in the cluster has remained over 75% for the last hour (including a list of which nodes are in trouble).

If you could show me how to build that alert, I could compare it with what I posted originally (which I largely got from the documentation for Watcher), to see how things are different now.

4

Hey,

so you can use the execute watch API... I created a small example and changed your query a bit

This is what my query looks like, it basically queries for the last hour and creates two filter aggs to agg by heap over and under 75%

GET .monitoring-es-*/node_stats/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "timestamp": {
              "gte": "now-1h",
              "lte": "now"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "minutes": {
      "date_histogram": {
        "field": "timestamp",
        "interval": "10m"
      },
      "aggs": {
        "nodes": {
          "terms": {
            "field": "source_node.name",
            "size": 10
          },
          "aggs": {
            "memory_over_75": {
              "filters": {
                "filters": {
                  "high": {
                    "range": {
                      "node_stats.jvm.mem.heap_used_percent": {
                        "gte": 75
                      }
                    }
                  },
                  "low": {
                    "range": {
                      "node_stats.jvm.mem.heap_used_percent": {
                        "lt": 75
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Now you cna use that output and emulate watch execution including script execution. One of the benefits of this is the fact that painless outputs pretty good error messages for a scripting language.

POST _xpack/watcher/watch/_execute
{
  "alternative_input": {
    "aggregations": {
      "minutes": {
        "buckets": [
          {
            "key_as_string": "2016-12-01T16:30:00.000Z",
            "key": 1480609800000,
            "doc_count": 27,
            "nodes": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "4re6dpX",
                  "doc_count": 27,
                  "memory_over_75": {
                    "buckets": {
                      "high": {
                        "doc_count": 0
                      },
                      "low": {
                        "doc_count": 27
                      }
                    }
                  }
                }
              ]
            }
          },
          {
            "key_as_string": "2016-12-01T16:40:00.000Z",
            "key": 1480610400000,
            "doc_count": 60,
            "nodes": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "4re6dpX",
                  "doc_count": 60,
                  "memory_over_75": {
                    "buckets": {
                      "high": {
                        "doc_count": 1
                      },
                      "low": {
                        "doc_count": 1
                      }
                    }
                  }
                }
              ]
            }
          }
        ]
      }
    }
  },
  "watch": {
    "trigger": {
      "schedule": {
        "interval": "10s"
      }
    },
    "input": {
      "simple": {}
    },
    "condition": {
      "script" : {
        "inline" : "return ctx.payload.aggregations.minutes.buckets.stream().anyMatch(b -> b.nodes.buckets.stream().anyMatch(x -> x.memory_over_75.buckets.high.doc_count > 0 && x.memory_over_75.buckets.low.doc_count == 0))"
      }
    },
    "actions": {
      "log_me": {
        "logging": {
          "text": "{{ctx}}"
        }
      }
    }
  }
}

painless resembles a very strict subset of java, so you might want to read about that.

--Alex

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.