Watch json error "Unknown key for a START_OBJECT in ["

Trying to define a watch alert in ELK to trigger when a file system reaches more than 87% full. I need to aggregate on host, files system and percent full for the FS. With the script below, I'm getting the error "Unknown key for a START_OBJECT in [aggs0].". I can't figure out what the unknown key is. All of the referenced fields in the aggs section exist and have been double checked. Any clue where I'm going wrong?

{
"trigger": {
"schedule": {
"interval": "24h"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"usis-oracle_dba_beats-*"
],
"rest_total_hits_as_int": true,
"body": {
"size": 0,
"query": {
"bool": {
"filter": {
"range": {
"@timestamp": {
"gte": "{{ctx.trigger.scheduled_time}}||-24h",
"lte": "{{ctx.trigger.scheduled_time}}",
"format": "strict_date_optional_time||epoch_millis"
}
}
}
}
},
"aggs0": {
"hostAgg": {
"terms": {
"field": "beat.hostname",
"size": "50"
},
"aggs1": {
"filesystemAgg": {
"terms": {
"field": "system.filesystem.mount_point",
"size": "50",
"order": {
"metricAgg": "desc"
}
},
"aggs": {
"metricAgg": {
"max": {
"field": "system.filesystem.used.pct"
}
}
}
}
}
}
}
}
}
}
},
"condition": {
"script": {
"source": "ArrayList arr = ctx.payload.aggregations.filesystemAgg.buckets; for (int i = 0; i < arr.length; i++) { if (arr[i]['metricAgg'].value > params.threshold) { return true; } } return false;",
"lang": "painless",
"params": {
"threshold": 85
}
}
},
"actions": {
"email_1": {
"email": {
"profile": "standard",
"to": [
"hubbard.gary@principal.com"
],
"subject": "Watcher [{{ctx.metadata.name}}] threshold reached for following databases: {{ctx.payload}}",
"body": {
"text": "Tablespaces {{ctx.payload}} crossed threshold of more than 85%. Check the dashboard for further diagnosis."
}
}
}
},
"transform": {
"script": {
"source": "HashMap result = new HashMap(); ArrayList arr = ctx.payload.aggregations.filesystemAgg.buckets; ArrayList filteredHits = new ArrayList(); for (int i = 0; i < arr.length; i++) { HashMap filteredHit = new HashMap(); filteredHit.key = arr[i].key; filteredHit.value = arr[i]['metricAgg'].value; if (filteredHit.value > params.threshold) { filteredHits.add(filteredHit); } } result.results = filteredHits; return result;",
"lang": "painless",
"params": {
"threshold": 85
}
}
}
}

Hey,

please take the time to properly format your snippets. This forums supports markdown, so it has excellent support for code snippets.

Also, sharing the whole exception, instead of the one line in the subject would help a lot.

In addition, information like your Elasticsearch version are important as well.

Also, quickly checking your condition, I do not think that it does, what you think it does. You try to iterate through a list of buckets, however you are already returning after checking the first bucket. So only a single bucket gets checked.

Hey Alexander,

My apologies. Hopefully this will have the information formatted properly. I did figure out part of my syntax issues and corrected them.

ES version: 7.7.0

I'm fairly new to Watcher, so I'm not surprised if what I have isn't quite doing what I wanted. I currently have an ES index that contains file system information for our Linux hosts. I need to have a watcher in place that will alert us when any of the hosts in the index have a specific file system that hits 87% full or more. I need to make sure the host name and file systems are identified in the email and/or other actions we choose so we know which host had whatever file system cross the threshold. Is that possible to do in one watcher definition, or do I have to create a watcher for each host name expected that is in the ES index? Having the host name, file system name and percent used identified in the actions is the only reason I attempted the aggregation that I did (although from an aggregation perspective it probably doesn't make much sense :slight_smile: ). Any clues and/or help is much appreciated.

    {
      "trigger": {
        "schedule": {
          "interval": "24h"
        }
      },
      "input": {
        "search": {
          "request": {
            "search_type": "query_then_fetch",
            "indices": [
              "usis-oracle_dba_beats-*"
            ],
            "rest_total_hits_as_int": true,
            "body": {
              "size": 0,
              "query": {
                "bool": {
                  "filter": {
                    "range": {
                      "@timestamp": {
                        "gte": "{{ctx.trigger.scheduled_time}}||-24h",
                        "lte": "{{ctx.trigger.scheduled_time}}",
                        "format": "strict_date_optional_time||epoch_millis"
                      }
                    }
                  }
                }
              },
              "aggs": {
                "hostAgg": {
                  "terms": {
                    "field": "beat.hostname",
                    "size": "50"
                  },
                  "aggs": {
                    "filesystemAgg": {
                      "terms": {
                        "field": "system.filesystem.mount_point",
                        "size": "50",
                        "order": {
                          "metricAgg": "desc"
                        }
                      },
                      "aggs": {
                        "metricAgg": {
                          "max": {
                            "field": "system.filesystem.used.pct"
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      },
      "condition": {
        "script": {
          "source": "ArrayList arr = ctx.payload.aggregations.filesystemAgg.buckets; for (int i = 0; i < arr.length; i++) { if (arr[i]['metricAgg'].value > params.threshold) { return true; } } return false;",
          "lang": "painless",
          "params": {
            "threshold": 0.87
          }
        }
      },
      "actions": {
        "email_1": {
          "email": {
            "profile": "standard",
            "to": [
              "hubbard.gary@principal.com"
            ],
            "subject": "Watcher [{{ctx.metadata.name}}] threshold reached for following databases: {{ctx.payload}}",
            "body": {
              "text": "Tablespaces {{ctx.payload}} crossed threshold of more than 87%. Check the dashboard for further diagnosis."
            }
          }
        }
      },
      "transform": {
        "script": {
          "source": "HashMap result = new HashMap(); ArrayList arr = ctx.payload.aggregations.filesystemAgg.buckets; ArrayList filteredHits = new ArrayList(); for (int i = 0; i < arr.length; i++) { HashMap filteredHit = new HashMap(); filteredHit.key = arr[i].key; filteredHit.value = arr[i]['metricAgg'].value; if (filteredHit.value > params.threshold) { filteredHits.add(filteredHit); } } result.results = filteredHits; return result;",
          "lang": "painless",
          "params": {
            "threshold": 0.87
          }
        }
      }
    }

Hey,

you have not specified the endpoint and the HTTP method you are sending this to. If I use the Put Watch API with your input on 7.9.2 this works for me, so there must be another difference.

Thank you for taking a look at it. I did get if figured out, so I'm on to the next thing. Maybe I should post this in another forum subject, but I have a question about the ctx.payload and what properties can be exposed. Can the data coming from the payload be formatted? For example, I'm getting back the ctx.payload and it provides a key and a value. The value is a number and since it's a percentage, the format is coming back with about 10 or 12 digits of precision. I would like to be able to format that down to two digits following the decimal. If I need to ask this under a different topic, I will. Just thought I'd through it out there.

Glad you got it working. For the next issue: You could use a transform and run String.format() in there.