Watcher payload json array access

kaushik.vankayala · November 22, 2018, 9:27am

Hi There,

I have created an advances watch/alert in Watcher, Kibana UI. I have an input which returns a payload that contains an array. I would like to access a specific field of all the elements in that array and add it in logging action or email or any other actions.

The query is;

GET mt_dev-experienceapp/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "ErrorMessage": "Invalid Response from Backend System" }},
        { "range": { "@timestamp": { "gte": "now-1h","lte": "now"}}
        }
        ]
    }
  }
}

This is the result of my query;

{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 3,
    "max_score": 2.4384103,
    "hits": [
      {
        "_index": "mt_dev-experienceapp",
        "_type": "doc",
        "_id": "P1Z7OmcBt4PCzwoDqDQq",
        "_score": 2.4384103,
        "_source": {
          "ErrorMessage": "Invalid Response from Backend System",
          "apiname": "/api/experience/citylist-airportcode/v1/citylist-airportcode",
          "loglevel": "ERROR",
          "audittype": "Audit-OUT",
          "PartnerID": "card",
          "host": "A2ML26095",
          "APITransactionId": "399844c3f4c64308a0ad868b9e4cb3a9",
          "app": "[[dev-experienceapp].HTTP_Experience_Listener_Configuration.worker.10]",
          "TimeStamp": "2018-11-21 11:43:17.763",
          "message": """[2018-11-22 06:52:28.211] ERROR   Audit-OUT [[dev-experienceapp].HTTP_Experience_Listener_Configuration.worker.10]: {"APITransactionId": "399844c3f4c64308a0ad868b9e4cb3a9", "StatusCode": "500","TimeStamp": "2018-11-21 11:43:17.763","PartnerID": "paytm","PartnerUniqueID": "5d13578905ff4c0aab7c5cb2cb6382c5","ErrorMessage": "Invalid Response from Backend System","Request": "/api/experience/citylist-airportcode/v1/citylist-airportcode","ResponseTime": "3373 ms"}""",
          "@timestamp": "2018-11-22T01:22:28.211Z",
          "PartnerUniqueID": "5d13578905ff4c0aab7c5cb2cb6382c5",
          "StatusCode": "500",
          "@version": "1",
          "ResponseTimeMilliSec": "3373"
        }
      },
      {
        "_index": "mt_dev-experienceapp",
        "_type": "doc",
        "_id": "xFYzOmcBt4PCzwoDZRRu",
        "_score": 1.9116077,
        "_source": {
          "ErrorMessage": "Invalid Response from Backend System",
          "apiname": "/api/experience/citylist-airportcode/v1/citylist-airportcode",
          "loglevel": "ERROR",
          "audittype": "Audit-OUT",
          "PartnerID": "wirecard",
          "host": "A2ML26095",
          "APITransactionId": "399844c3f4c64308a0ad868b9e4cb3a9",
          "app": "[[dev-experienceapp].HTTP_Experience_Listener_Configuration.worker.10]",
          "TimeStamp": "2018-11-21 11:43:17.763",
          "message": """[2018-11-22 06:52:28.211] ERROR   Audit-OUT [[dev-experienceapp].HTTP_Experience_Listener_Configuration.worker.10]: {"APITransactionId": "399844c3f4c64308a0ad868b9e4cb3a9", "StatusCode": "500","TimeStamp": "2018-11-21 11:43:17.763","PartnerID": "payback","PartnerUniqueID": "5d032ba3a5ff4c0aab7c5cb2cb6382c5","ErrorMessage": "Invalid Response from Backend System","Request": "/api/experience/citylist-airportcode/v1/citylist-airportcode","ResponseTime": "3373 ms"}""",
          "@timestamp": "2018-11-22T01:22:28.211Z",
          "PartnerUniqueID": "5d032ba3a5ff4c0aab7c5cb2cb6382c5",
          "StatusCode": "500",
          "@version": "1",
          "ResponseTimeMilliSec": "3373"
        }
      },
      {
        "_index": "mt_dev-experienceapp",
        "_type": "doc",
        "_id": "-1Z_OmcBt4PCzwoDlzUd",
        "_score": 1.5268025,
        "_source": {
          "ErrorMessage": "Invalid Response from Backend System",
          "apiname": "/api/experience/citylist-airportcode/v1/citylist-airportcode",
          "loglevel": "ERROR",
          "audittype": "Audit-OUT",
          "PartnerID": "bookmyshow",
          "host": "A2ML26095",
          "APITransactionId": "399844c3f4c64308a0ad868b9e4cb3a9",
          "app": "[[dev-experienceapp].HTTP_Experience_Listener_Configuration.worker.10]",
          "TimeStamp": "2018-11-21 11:43:17.763",
          "message": """[2018-11-22 13:45:28.211] ERROR   Audit-OUT [[dev-experienceapp].HTTP_Experience_Listener_Configuration.worker.10]: {"APITransactionId": "399844c3f4c64308a0ad868b9e4cb3a9", "StatusCode": "500","TimeStamp": "2018-11-21 11:43:17.763","PartnerID": "wirecard","PartnerUniqueID": "5d032ba3a5ff4c0aab7c5cb2cb6382c5","ErrorMessage": "Invalid Response from Backend System","Request": "/api/experience/citylist-airportcode/v1/citylist-airportcode","ResponseTime": "3373 ms"}""",
          "@timestamp": "2018-11-22T08:15:28.211Z",
          "PartnerUniqueID": "5d032ba3a5ff4c0aab7c5cb2cb6382c5",
          "StatusCode": "500",
          "@version": "1",
          "ResponseTimeMilliSec": "3373"
        }
      }
    ]
  }
}

So in the hits i have 3 results. I would like to display all the partners in the logging oe email action text data.

The thing is the hits.total is not a fixed value as it depends on number of results obtained. So how do i write the logging action or email action to get all the partnerids...

Below is my current watcher configuration, but looking for dynamically obtain the partners list ;

{
  "trigger": {
    "schedule": {
      "interval": "1h"
    }
  },
  "input": {
    /*clipped off as the topic limit exceeded maximum*/
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gte": 0
      }
    }
  },
  "actions": {
    "my-logging-action": {
      "logging": {
        "level": "info",
        "text": "There are {{ctx.payload.hits.total}} documents in your index for Backend System Errors. The recent one is from the partner {{ctx.payload.hits.hits.0._source.PartnerID}} "
      }
    }
  }
}

Any help, is greatly appreciated!

Regards

Kaushik

Marius_Dragomir · November 22, 2018, 12:19pm

The scripting language used here is Mustache and that is a logicless language, so it is impossible to iterate through an array with it. What you're doing currently is pretty much the most you can accomplish. But, there are improvements to the way watches work in the pipeline so that will make it easier to define more complex actions.

kaushik.vankayala · November 22, 2018, 2:02pm

Thanks @Marius_Dragomir for your reply. I have few more doubts stated as below;
I see there is a scripting language called painless. Can that be used in action part of watch? If not, at least i see it can be used in condition section of watcher. That leads me to another doubt (which i could not find in documentation) is it possible have a variable defined there and access them in the text part of logging or body part of email action?

What did you mean by watches in pipeline? Can you gimme some references or suggestions as to how i could achieve the above requirement? (to list all the partners resulted in a query?)

kaushik.vankayala · November 28, 2018, 10:31am

I solved this question by following one of the discussions that is listed below!

This is where we can get the complete manual for Mustache language description

https://mustache.github.io/mustache.5.html

Hope it helps someone!

Regards

Kaushik

system · December 26, 2018, 10:31am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.