Can't print string values from groovy script

Elastic Stack Elasticsearch
1

Hello,

I try to print browser use agent values from aggregated bucket:

"ctx.payload.aggregations.ua_buckets.buckets": [
           { "doc_count": 11,
             "key": "Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.0;+Trident/4.0;+Acoo+Browser;+GTB5;+Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1)+;+InfoPath.1;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30618)"
           },
           {
             "doc_count": 11,
             "key": "Mozilla/5.0+(compatible;+MSIE+8.0;+Windows+NT+6.0;+Trident/4.0;+Acoo+Browser+1.98.744;+.NET+CLR+3.5.30729)"
           },

with the following script:

{
  "script": "return [ myhost: ctx.payload.aggregations.ua_buckets.buckets.collect { [ myua:it.key, mycount:it.doc_count ] } ]"
}

I also tried myua:it.key_as_string

However, values are nulls:

"actions": [
        {
          "id": "notify_log",
          "type": "logging",
          "status": "success",
          "transform": {
            "type": "script",
            "status": "success",
            "payload": {
              "myhost": [
                {
                  "myua": null,
                  "mycount": 11
                },
                {
                  "myua": null,
                  "mycount": 11
                },

"logging": {
"logged_text": "User Agent - # of attempts

  • 11
  • 11
  • 11
  • 10
  • 10
  • 10
  • 8
  • 7
  • 7
  • 7
    "

Please advise

2

Hey,

first guess: keyis a keyword in the groovy language and thus not defined. Can you try it["key"]?

--Alex

3

Alex,

To pour more light to the issue, let me show complete watchers here. One of them is intended to print list of IPs, and it works flawlessly. Another one has the exact same semantic , in exception that it should print user agents. And this does not work.

Please see working watcher in my next reply below. And here is non-working watcher for "user agent":

{
    "trigger": {
      "schedule": {
        "interval": "1m"
      }
    },
    "input": {
      "search": {
        "request": {
          "search_type": "query_then_fetch",
          "indices": [
            "example-*"
          ],
          "types": [

          ],
          "body": {
            "size": 0,
            "query": {
              "bool": {
                "must": [
                  {
                    "range": {
                      "date": {
                        "gte": "now-1h",
                        "lt": "now"
                      }
                    }
                  },
                  {
                    "terms": {
                      "cs_uri_stem": [
                        "/myaccount",
                        "/user/login"
                      ],
                      "execution": "or"
                    }
                  },
                  {
                    "terms": {
                      "cs_method": [
                        "post",
                        "put"
                      ],
                      "execution": "or"
                    }
                  }
                ]
              }
            },
            "aggs": {
              "ua_buckets": {
                "terms": {
                  "field": "cs(User_Agent)",
                  "min_doc_count": 5
                }
              }
            }
          }
        }
      }
    },
    "condition": {
      "array_compare": {
        "ctx.payload.aggregations.ua_buckets.buckets": {
          "path": "doc_count",
          "gte": {
            "value": 1,
            "quantifier": "some"
          }
        }
      }
    },
    "actions": {
      "send_email": {
        "throttle_period": "30m",
        "transform": {
          "script": {
            "id": "indexedCollectDataFromBucketsUserAgent",
            "lang": "groovy"
          }
        },
        "email": {
          "profile": "standard",
          "to": [
            "myteam@example.com"
          ],
          "subject": "ElasticSearch Cluster Alert -- Too many login attempts were made from with particular user agent during last 60 min",
          "body": {
            "text": "UserAgent - # of attempts\n{{#ctx.payload.myhost}}\n{{myua}} - {{mycount}}\n{{/ctx.payload.myhost}}"
          }
        }
      },
      "notify_slack": {
        "throttle_period": "30m",
        "transform": {
          "script": {
            "id": "indexedCollectDataFromBucketsUserAgent",
            "lang": "groovy"
          }
        },
        "slack": {
          "message": {
            "to": [
              "#monitoring"
            ],
            "text": "ElasticSearch Watcher",
            "attachments": [
              {
                "color": "warning",
                "title": "Too many login attempts were made with particular user agent during last 60 min",
                "text": "User Agent - # of attempts\n{{#ctx.payload.myhost}}\n{{myua}} - {{mycount}}\n{{/ctx.payload.myhost}}"
              }
            ]
          }
        }
      },
      "notify_log": {
        "throttle_period": "1m",
        "transform": {
          "script": {
            "id": "indexedCollectDataFromBucketsUserAgent",
            "lang": "groovy"
          }
        },
        "logging": {
          "level": "info",
          "text": "User Agent - # of attempts\n{{#ctx.payload.myhost}}\n{{myua}} - {{mycount}}\n{{/ctx.payload.myhost}}"
        }
      }
    }
  }

Here is the groovy script as well:

{
"lang": "groovy",
"_id": "indexedCollectDataFromBucketsUserAgent",
"found": true,
"_version": 4,
"script": "return [ myhost: ctx.payload.aggregations.ua_buckets.buckets.collect { [ myua:it.key_as_string, mycount:it.doc_count ] } ]"
}

Please evaluate

4

The working watcher for IPs:

  {
        "trigger": {
          "schedule": {
            "interval": "1m"
          }
        },
        "input": {
          "search": {
            "request": {
              "search_type": "query_then_fetch",
              "indices": [
                "example-*"
              ],
              "types": [

              ],
              "body": {
                "size": 0,
                "query": {
                  "bool": {
                    "must": [
                      {
                        "range": {
                          "date": {
                            "gte": "now-1h",
                            "lt": "now"
                          }
                        }
                      },
                      {
                        "terms": {
                          "cs_uri_stem": [
                            "/myaccount",
                            "/user/login"
                          ],
                          "execution": "or"
                        }
                      },
                      {
                        "terms": {
                          "cs_method": [
                            "post",
                            "put"
                          ],
                          "execution": "or"
                        }
                      }
                    ]
                  }
                },
                "aggs": {
                  "ip_buckets": {
                    "terms": {
                      "field": "c_ip",
                      "min_doc_count": 5
                    }
                  }
                }
              }
            }
          }
        },
        "condition": {
          "array_compare": {
            "ctx.payload.aggregations.ip_buckets.buckets": {
              "path": "doc_count",
              "gte": {
                "value": 25,
                "quantifier": "some"
              }
            }
          }
        },
        "actions": {
          "send_email": {
            "throttle_period": "30m",
            "transform": {
              "script": {
                "id": "indexedCollectDataFromBuckets",
                "lang": "groovy"
              }
            },
            "email": {
              "profile": "standard",
              "to": [
                "myteam@example.com"
              ],
              "subject": "ElasticSearch Cluster Alert -- Too many login attempts were made from particular IP address(-es) during last 60 min",
              "body": {
                "text": "IP address - # of attempts\n{{#ctx.payload.myhost}}\n{{myip}} - {{mycount}}\n{{/ctx.payload.myhost}}"
              }
            }
          },
          "notify_slack": {
            "throttle_period": "30m",
            "transform": {
              "script": {
                "id": "indexedCollectDataFromBuckets",
                "lang": "groovy"
              }
            },
            "slack": {
              "message": {
                "to": [
                  "#monitoring"
                ],
                "text": "ElasticSearch Watcher",
                "attachments": [
                  {
                    "color": "warning",
                    "title": "Too many login attempts were made from particular IP address(-es) during last 60 min",
                    "text": "IP address - # of attempts\n{{#ctx.payload.myhost}}\n{{myip}} - {{mycount}}\n{{/ctx.payload.myhost}}"
                  }
                ]
              }
            }
          }
        }
      }

with groovy script:

{
  "lang": "groovy",
  "_id": "indexedCollectDataFromBuckets",
  "found": true,
  "_version": 1,
  "script": "return [ myhost: ctx.payload.aggregations.ip_buckets.buckets.collect { [ myip:it.key_as_string, mycount:it.doc_count ] } ]"
}
5

Alex pointed me to the right direction. The issue was with incorrect string parsing.
My fix is to add "$" to the beginning of the variable, like this:

"notify_log": {
        "throttle_period": "1m",
        "transform": {
          "script": {
            "id": "indexedCollectDataFromBucketsUserAgent",
            "lang": "groovy"
          }
        },
        "logging": {
          "level": "info",
          "text": "User Agent - # of attempts\n{{#ctx.payload.myhost}}\n${{myua}} - {{mycount}}\n{{/ctx.payload.myhost}}"
        }
      }
    }