Watch Marvel node_left event. Requesting fields from .marvel-* search

Elastic Stack Elasticsearch
1

Hello again!
Below watch:

> {
> "trigger": {
>     "schedule": {
>       "interval": "60s"
>     }
>   },
>   "input": {
>     "search": {
>       "request": {
>         "indices": [
>           ".marvel-*"
>         ],
>         "search_type": "query_then_fetch",
>         "body": {
>           "query": {
>             "filtered": {
>               "query": {
>                 "bool": {
>                   "should": [
>                     {
>                       "match": {
>                         "event": "node_left"
>                       }
>                     }
>                   ]
>                 }
>               },
>               "filter": {
>                 "range": {
>                   "@timestamp": {
>                     "from": "{{ctx.trigger.scheduled_time}}||-60s",
>                     "to": "{{ctx.trigger.triggered_time}}"
>                   }
>                 }
>               }
>             }
>           },
>           "fields": [
>             "event",
>             "node.name",
>             "node.ip_port",
>             "message",
>             "cluster_name"
>           ],
>           "sort": [
>             {
>               "@timestamp": {
>                 "order": "desc"
>               }
>             }
>           ]
>         }
>       }
>     }
>   },
>   "throttle_period": "60s",
>   "condition": {
>     "script": {
>       "inline": "ctx.payload.hits.size() > 0 "
>     }
>   },
>   "actions": {
>     "send_email": { 
>       "email": {
>         "to": "elkreceive@gmail.com", 
>         "subject": " Watcher Notification - Event: {{ctx.payload.hits.hits.0.fields.node.name}}{{ctx.payload.hits.hits.0.fields.node.ip_port}}{{ctx.payload.hits.hits.0.fields.message}} the cluster at {{ctx.trigger.triggered_time}}",
>         "body": "{{ctx.payload.hits.hits.0.fields.message}}{{ctx.payload.hits.hits.0.fields.node.name}}{{ctx.payload.hits.hits.0.fields.node.ip_port}} the cluster {{ctx.payload.hits.hits.0.fields.cluster_name}} "
>       }
>     }
>   }
> }

Resulting mail returns:

> Watcher Notification - Event: {0=[esclient1][10.0.0.202:9300] joined} the cluster at 2015-09-16T11:49:34.978Z

Notice fields.message and fields.cluster_name return their values. However node.name and node.ip_port do not. I have also tried {{ctx.payload.hits.hits.0._source.node.name}} yet these are returned empty.

The .marvel-* indice in kibana shows these terms filled with the desired values.

Does anyone have any idea where I have made mistakes?

2

Update:
If I grab a watch in .watch_history*, I get the following: > "node.name":["esclient1"],"event":["node_left"],"node.ip_port":["10.0.0.202:9300"] So the watch is grabbing the desired fields. They are just not being placed in the mail. So the problem is in:

"subject": " Watcher Notification - Event: {{ctx.payload.hits.hits.0.fields.node.name}}{{ctx.payload.hits.hits.0.fields.node.ip_port}}{{ctx.payload.hits.hits.0.fields.message}} the cluster at {{ctx.trigger.triggered_time}}", "body": "{{ctx.payload.hits.hits.0.fields.message}}{{ctx.payload.hits.hits.0.fields.node.name}}{{ctx.payload.hits.hits.0.fields.node.ip_port}} the cluster {{ctx.payload.hits.hits.0.fields.cluster_name}} "

Another issue I've noticed is that {{ctx.trigger.triggered_time}} shows UTC even when timezones have been adjusted and even after adding a timezone filter to logstash. Kibana seems to adjust the timestamp when indexing to the correct time, but it seems this does not happen when sending watch triggered mails.

3

Update:
Still have not been able to find anything to get this working. Does anyone have an idea?

4

If you use _source instead fields then I think it should work. So for example:

{{ctx.payload.hits.hits.0._source.node.name}}
5

Hi, afraid it does not.

{
"trigger": {
    "schedule": {
      "interval": "60s"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": [
          ".marvel-*"
        ],
        "search_type": "query_then_fetch",
        "body": {
          "query": {
            "filtered": {
              "query": {
                "bool": {
                  "should": [
                    {
                      "match": {
                        "event": "node_left"
                      }
                    }
                  ]
                }
              },
              "filter": {
                "range": {
                  "@timestamp": {
                    "from": "{{ctx.trigger.scheduled_time}}||-60s",
                    "to": "{{ctx.trigger.triggered_time}}"
                  }
                }
              }
            }
          },
          "fields": [
            "event",
            "node.name",
            "node.ip_port",
            "message",
            "cluster_name",
            "node.attributes.data",
            "node.attributes.master",
            "@timestamp"
          ],
          "sort": [
            {
              "@timestamp": {
                "order": "desc"
              }
            }
          ]
        }
      }
    }
  },
  "throttle_period": "60s",
  "condition": {
    "script": {
      "inline": "ctx.payload.hits.size() > 0 "
    }
  },
  "actions": {
    "send_email": { 
      "email": {
        "to": "elkreceive@gmail.com", 
        "subject": " Watcher Notification - Event: NODE:{{ctx.payload.hits.hits.0._source.node.name}} ON IP:{{ctx.payload.hits.hits.0._source.node.ip_port}} MESSAGE:{{ctx.payload.hits.hits.0.fields.message}} the cluster at {{ctx.trigger.triggered_time}} UTC",
        "body": { 
        "html": "<HTML><b>Trigger time: </b>{{ctx.trigger.triggered_time}} UTC<br><b>Node: </b>{{ctx.payload.hits.hits.0._source.node.name}}<br><b>IP and Port: </b>{{ctx.payload.hits.hits.0._source.node.ip_port}}<br><b>Master node: </b>{{ctx.payload.hits.hits.0._source.node.attributes.master}}<br><b>Data node: </b>{{ctx.payload.hits.hits.0._source.node.attributes.data}}<br><b>Message: </b>{{ctx.payload.hits.hits.0.fields.message}}<br><b>Cluster: </b>{{ctx.payload.hits.hits.0.fields.cluster_name}}</HTML>"
        }
      }
    }
  }
}

Results in empty returns:

Wish the documentation would go into a bit more detail.

6

Got it working with:

          "fields": [
            "event",
            "message",
            "cluster_name",
            "@timestamp"
          ],
           "_source": [
            "node.name",
            "node.ip_port",
            "node.attributes.data",
            "node.attributes.master"
          ],
7

Ah I missed that you specifically asking for fields. The fields option should only be used if you configure in the mappings that you store fields. If fields was left out of the search request the _source would be able to pick your values up. By default the entire _source of a hit is returned. (but that isn't the case if fields is used in the search request)

8

Hi guys,

I´ve tried the example watch, but in my Marvel indices aren't an event field. Someone has an idea?