Cannot figure out how to get data from _source into Watch Output

Elastic Stack Elasticsearch
1

I am having a challenge with obtaining data from my original source of the Alert to go in the output. I have used the extract and ctx.payload.hits.hits.0._source syntax that I have found on searching but no go. Can someone please look this over and let me know what I am doing wrong? My goal is to get a field called "fi_instance" into the output. From this I hope to hit a webhook with the source data and POST it. This alert is simply using logging for testing. Thanks in advance.

{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "filebeat*"
        ],
        "types": [],
        "body": {
          "query": {
            "exists": {
              "field": "error_data"
            }
          }
        }
      },
      "extract": [
        "rt_account_status",
        "fi_instance",
        "fetch_time"
      ]
    }
  },
  "condition": {
    "compare": {
      "ctx.execution_time": {
        "gte": "<{now-5m}>"
      }
    }
  },
  "actions": {
    "my-logging-action": {
      "logging": {
        "level": "info",
        "text": "{{ctx.payload.hits.hits.0._source.fi_instance}} had an error of {{ctx.payload.error_data}} at {{ctx.execution_time}}"
      }
    }
  }
}
1 Like
2

Hey,

two things here:

  • First, the extraction does not work, because it is referring to the full document and not only inside the _source field. You can just omit it.
  • Second, your condition will always match, because the execution time of the watch has nothing in common with the date filter of your search query (right now there is none, but this is where you should filter) - then you can check in the condition if the number of hits is bigger than zero.

Let's get up and running with a useful workflow for debugging this. Instead of storing the watch, you should use the Execute Watch API like this

# lets create a document matching our query
PUT filebeat-foo/foo/1
{
  "error_data" : "something",
  "rt_account_status" : "rt_account_status",
  "fi_instance" : "fi_instance",
  "fetch_time" : 1234
}

# ensure it matches
GET filebeat-foo/_search
{
  "query": {
    "exists": {
      "field": "error_data"
    }
  }
}

# lets write a fancy watch
PUT _xpack/watcher/watch/_execute
{
  "watch": {
    "trigger": {
      "schedule": {
        "interval": "1m"
      }
    },
    "input": {
      "search": {
        "request": {
          "search_type": "query_then_fetch",
          "indices": [
            "filebeat*"
          ],
          "types": [],
          "body": {
            "query": {
              "exists": {
                "field": "error_data"
              }
            }
          }
        }
      }
    },
    "condition": {
      "compare": {
        "ctx.execution_time": {
          "gte": "<{now-5m}>"
        }
      }
    },
    "actions": {
      "my-logging-action": {
        "logging": {
          "level": "info",
          "text": "{{ctx.payload.hits.hits.0._source.fi_instance}} had an error of {{ctx.payload.error_data}} at {{ctx.execution_time}}"
        }
      }
    }
  }
}

This way you can easily modify your watch and check out the response in real time, no need to wait for anything.

In addition you might want to add aggregations in the query, so you can get the counts of each unique field value of the fi_instance field instead of checking the value of each response, but lets get started slowly here...

Hope this helps!

--Alex

3

Thanks @spinscale that helps a lot. I still get a strange output for Logging (see below) but honestly seeing the results this time versus not really helps. I think I will index the results and see what i can do with it versus using the Logging output. Webhook and Index will be my major outputs anyway.

You can see here that the ctx.payload.hits.hits.0._source.fi_instance even with your code exactly never seems to get placed in the output thus my frustration. I will try the other actions to see if I can echo the results as that is entirely what I need for the next part of our workflow.

"actions": [
  {
    "id": "my-logging-action",
    "type": "logging",
    "status": "success",
    "logging": {
      "logged_text": " had an error of  at 2017-05-22T14:15:45.418Z"
    }
  }
4

Hey,

just execute the query (without watcher) and check the response of the first hit and if it includes that field. I suppose it is not there, maybe because you also have different documents in that indx, that match your query?

--Alex

5

The field is there in the results but with multiple results and a single logging message that one to many may simply be the issue.

Is this the best location for examples of more advanced watchers? If you have any other recommended sites or places please let me know. Thanks so much. https://github.com/elastic/examples

6

Hey,

if you copy/paste the output of your execute watch API, debugging would be much easier :slight_smile:

And yeah, that's a good spot to take a look for further alerting examples.

--Alex

7

Here is that output.....had to remove some info since it is sensitive so you may see a lot of "xxx" and "xxxx". Also to save space I removed a lot of the results as there were 240 of them. Thanks for your input.

{
  "watch_id": "Errors",
  "state": "executed",
  "_status": {
    "state": {
      "active": true,
      "timestamp": "2017-05-22T14:12:45.110Z"
    },
    "last_checked": "2017-05-22T14:51:45.250Z",
    "last_met_condition": "2017-05-22T14:51:45.250Z",
    "actions": {
      "my-logging-action": {
        "ack": {
          "timestamp": "2017-05-22T14:13:45.372Z",
          "state": "ackable"
        },
        "last_execution": {
          "timestamp": "2017-05-22T14:51:45.250Z",
          "successful": true
        },
        "last_successful_execution": {
          "timestamp": "2017-05-22T14:51:45.250Z",
          "successful": true
        }
      }
    }
  },
  "trigger_event": {
    "type": "schedule",
    "triggered_time": "2017-05-22T14:51:45.250Z",
    "schedule": {
      "scheduled_time": "2017-05-22T14:51:45.054Z"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "filebeat*"
        ],
        "types": [],
        "body": {
          "query": {
            "exists": {
              "field": "error_data"
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.execution_time": {
        "gte": "<{now-5m}>"
      }
    }
  },
  "metadata": {
    "name": "Error Alerts"
  },
  "result": {
    "execution_time": "2017-05-22T14:51:45.250Z",
    "execution_duration": 31,
    "input": {
      "type": "search",
      "status": "success",
      "payload": {
        "_shards": {
          "total": 240,
          "failed": 0,
          "successful": 240
        },
        "hits": {
          "hits": [
            {
              "_index": "filebeat-2017.04.05",
              "_type": "log",
              "_source": {
                "offset": 49936,
                "target_url": "https://xxx",
                "input_type": "log",
                "source": "/var/local/xxx/yyy_yyy40.log.04-05-2017",
                "message": "04/05/2017 05:02:33 instance|9999999\tfetch_time|31.614\ttotal_accounts|\trt_accounts|\trt_account_status|Fail\trt_time_acct|\terror_data|Unable to extract data\ttarget_url|https://xxx\tuserlogon|xxx",
                "type": "log",
                "tags": [
                  "beats_input_codec_plain_applied",
                  "i_9999999",
                  "fi_xxxx_xxxx40"
                ],
                "fetch_time": "31.614",
                "instance_num": "9999999",
                "@timestamp": "2017-04-05T10:02:33.000Z",
                "@version": "1",
                "beat": {
                  "hostname": "xxx.local",
                  "name": "xxx.local",
                  "version": "5.4.0"
                },
                "host": "xxx.local",
                "rt_account_status": "Fail",
                "fi_endpoint": "xxx_xxxx40",
                "timestamp": "04/05/2017 05:02:33",
                "error_data": "Unable to extract data"
              },
              "_id": "AVwTSYCn9e7I6Rr1_O2v",
              "_score": 1
            },
            {
              "_index": "filebeat-2017.04.05",
              "_type": "log",
              "_source": {
                "offset": 171304,
                "target_url": "https://xxxx",
                "input_type": "log",
                "source": "/var/local/xxx/xxxx_xxxx40.log.04-05-2017",
                "message": "04/05/2017 17:10:33 instance|9999998\tfetch_time|31.255\ttotal_accounts|\trt_accounts|\trt_account_status|Fail\trt_time_acct|\terror_data|Unable to extract data\ttarget_url|https://xxx\tuserlogon|xxx",
                "type": "log",
                "tags": [
                  "beats_input_codec_plain_applied",
                  "i_9999998",
                  "fi_xxxx_xxxx40"
                ],
                "fetch_time": "31.255",
                "instance_num": "9999998",
                "@timestamp": "2017-04-05T22:10:33.000Z",
                "@version": "1",
                "beat": {
                  "hostname": "xxx.local",
                  "name": "xxx.local",
                  "version": "5.4.0"
                },
                "host": "xxx.local",
                "rt_account_status": "Fail",
                "fi_endpoint": "xxxx_xxxx40",
                "timestamp": "04/05/2017 17:10:33",
                "error_data": "Unable to extract account overview data"
              },
              "_id": "AVwTUf0v9e7I6Rr1_RuS",
              "_score": 1
            }
          ],
          "total": 174881,
          "max_score": 1
        },
        "took": 29,
        "timed_out": false
      },
      "search": {
        "request": {
          "search_type": "query_then_fetch",
          "indices": [
            "filebeat*"
          ],
          "types": [],
          "body": {
            "query": {
              "exists": {
                "field": "error_data"
              }
            }
          }
        }
      }
    },
    "condition": {
      "type": "compare",
      "status": "success",
      "met": true,
      "compare": {
        "resolved_values": {
          "ctx.execution_time": "2017-05-22T14:51:45.250Z",
          "now-5m": "2017-05-22T14:46:45.295Z"
        }
      }
    },
    "actions": [
      {
        "id": "my-logging-action",
        "type": "logging",
        "status": "success",
        "logging": {
          "logged_text": " had an error of  at 2017-05-22T14:51:45.250Z"
        }
      }
    ]
  },
  "messages": []
}
8

@spinscale If you have an example of taking this payload of results and hitting a webhook and/or consolidating into a single index hit that would be amazing. I will move this from 5min to 1min checks most likely to lessen the result set.

9

This is the snippet of your first search hit. It does not contain a field named fi_instance which you referred to your first watch - that would explain why the logged text is empty at that point.

Just to repeat my point: Your condition in its current state is useless. You need to put the time into your query, otherwise you will get the same results over and over again with each execution, once your query matches once. All the examples in the examples repo contain a timestamp in the query and only check for the number of hits in the condition, so you should check them out.

Also I still think you should play around with aggregations in your search, like one of the examples in the docs.

If you want to put documents into another index, you can consider the index action instead of the webhook. If you transform your payload as part of that action, you can fill up a field named _doc as an array and you could index each document into a new index.

--Alex

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.