# Issue with integrating watcher webhook with ServiceNow

**URL:** <https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-alerting\
**Created:** [December 11, 2019, 10:35pm UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536 "2019-12-11T22:35:36Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![lcui\_dxc](https://avatars.discourse-cdn.com/v4/letter/l/8edcca/32.png) [@lcui\_dxc](https://discuss.elastic.co/u/lcui_dxc)\
**Post date:** [December 11, 2019, 10:35pm UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536/1 "2019-12-11T22:35:36Z")

</div>

Hello there:

We face issue when integrating watcher webhook to integrate with ServiceNow.  
Here is what we did and got…. If possible we’d prefer to set up a session so I may show you the details and learn from you.

We tested using Curl on linux commandline on kibana Server, we were able to create incidents on ServiceNow (working fine we can see the incidents there)  
We tested using POSTMan, it works fine…

Here is how we set up the webhook...

 ![watcher_issue](https://us1.discourse-cdn.com/elastic/original/3X/a/f/af28a65a6c1661a2d61618c8684c52cd127f2b1a.png)

Here is how we set up the webhook.

{  
"EventList": [  
{  
"severity": "Minor",  
"incidentImpact": 4,  
"title": "IPopov2TestELK5",  
"node": "PdxcTestCI-DoNotDelete-3",  
"eventsourcesendingserver": "testhostRD2ELK1",  
"eventsourcebacksyncserver": "",  
"eventsourceexternalid": "testhostRD2ELK1",  
"eventsourcecreatedtime": "2019-12-11-09-20-38",  
"category": "OOSS",  
"application": "APP\_NAME\_RD2ELK1",  
"incidentCategory": "SoftwaretestRD2ELK1",  
"incidentSubcategory": "ApplicationtestRD2ELK1"  
}  
]  
}

////////////////////////////

If we use port 443, we got “received [400] status code”

The kibana log generated following event when use port 443. The method we used was “POST” but it seems in the log it shows as “PUT”.

{"type":"response","@timestamp":"2019-12-11T19:49:08Z","tags":,"pid":93202,"method":"put","statusCode":200,"req":{"url":"/api/watcher/watch/execute","method":"put","headers":{"host":"104.46.57.143:5601","connection":"keep-alive","content-length":"1503","accept":"application/json, text/plain, _/_","origin":"[http://logstash.143:5601](http://logstash.143:5601)","kbn-version":"7.3.2","user-agent":"Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"[http://104.46.57.143:5601/app/kibana","accept-encoding":"gzip](http://104.46.57.143:5601/app/kibana%22,%22accept-encoding%22:%22gzip), deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"24.14.147.207","userAgent":"24.14.147.207","referer":"[http://104.46.57.143:5601/app/kibana"},"res":{"statusCode":200,"responseTime":63,"contentLength":9},"message":"PUT](http://104.46.57.143:5601/app/kibana%22%7D,%22res%22:%7B%22statusCode%22:200,%22responseTime%22:63,%22contentLength%22:9%7D,%22message%22:%22PUT) /api/watcher/watch/execute 200 63ms - 9.0B"}

If we use port 80, we got  
"Sample request sent to [api.xxxx-d0.com:80/eve3-dev/yyy/events/R1/create](http://api.xxxx-d0.com:80/eve3-dev/yyy/events/R1/create)"

It seems the request was sent to ServiceNow, but it didn't show up on ServiceNow. The ServiceNow Admin checked and said the request even didn't reach to servicenow.

The kibana log generated following event when use port 443. The method we used was “POST” but it seems in the log it shows as “PUT”. Both entries in the kibana log are same…

{"type":"response","@timestamp":"2019-12-11T19:49:30Z","tags":,"pid":93202,"method":"put","statusCode":200,"req":{"url":"/api/watcher/watch/execute","method":"put","headers":{"host":"104.46.57.143:5601","connection":"keep-alive","content-length":"1501","accept":"application/json, text/plain, _/_","origin":"[http://logstash:5601](http://logstash:5601)","kbn-version":"7.3.2","user-agent":"Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"[http://104.46.57.143:5601/app/kibana","accept-encoding":"gzip](http://104.46.57.143:5601/app/kibana%22,%22accept-encoding%22:%22gzip), deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"24.14.147.207","userAgent":"24.14.147.207","referer":"[http://104.46.57.143:5601/app/kibana"},"res":{"statusCode":200,"responseTime":30,"contentLength":9},"message":"PUT](http://104.46.57.143:5601/app/kibana%22%7D,%22res%22:%7B%22statusCode%22:200,%22responseTime%22:30,%22contentLength%22:9%7D,%22message%22:%22PUT) /api/watcher/watch/execute 200 30ms - 9.0B"}

Any help would be highly appreciated...

Thanks

Li

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [December 17, 2019, 12:26pm UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536/2 "2019-12-17T12:26:09Z")

</div>

can you share the full watch and the output of the execute watch API in a gist?

---

<div class="post-metadata">

**Author:** ![lcui\_dxc](https://avatars.discourse-cdn.com/v4/letter/l/8edcca/32.png) [@lcui\_dxc](https://discuss.elastic.co/u/lcui_dxc)\
**Post date:** [December 18, 2019, 6:15pm UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536/3 "2019-12-18T18:15:17Z")

</div>

For this we created the watch on Kibana as showing about.

---

<div class="post-metadata">

**Author:** ![ipopov2](https://avatars.discourse-cdn.com/v4/letter/i/d26b3c/32.png) [@ipopov2](https://discuss.elastic.co/u/ipopov2)\
**Post date:** [December 19, 2019, 7:33am UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536/4 "2019-12-19T07:33:19Z")

</div>

Hi spinscale, Li, here are the watcher settings:

```
{
"found": true,
"_id": "78c7e199-2f05-4975-a907-a2f5e8408fb1",
"_version": 864,
"_seq_no": 229920,
"_primary_term": 9,
"status": {
    "state": {
        "active": true,
        "timestamp": "2019-12-13T07:20:00.500Z"
    },
    "last_checked": "2019-12-19T07:21:56.464Z",
    "actions": {
        "webhook_1": {
            "ack": {
                "timestamp": "2019-12-13T07:20:00.500Z",
                "state": "awaits_successful_execution"
            }
        }
    },
    "execution_state": "execution_not_needed",
    "version": 864
},
"watch": {
    "trigger": {
        "schedule": {
            "interval": "10m"
        }
    },
    "input": {
        "search": {
            "request": {
                "search_type": "query_then_fetch",
                "indices": [
                    "metricbeat-*"
                ],
                "rest_total_hits_as_int": true,
                "body": {
                    "size": 0,
                    "query": {
                        "bool": {
                            "filter": {
                                "range": {
                                    "@timestamp": {
                                        "gte": "{{ctx.trigger.scheduled_time}}||-5m",
                                        "lte": "{{ctx.trigger.scheduled_time}}",
                                        "format": "strict_date_optional_time||epoch_millis"
                                    }
                                }
                            }
                        }
                    },
                    "aggs": {
                        "metricAgg": {
                            "min": {
                                "field": "system.process.cpu.total.pct"
                            }
                        }
                    }
                }
            }
        }
    },
    "condition": {
        "script": {
            "source": "if (ctx.payload.aggregations.metricAgg.value > params.threshold) { return true; } return false;",
            "lang": "painless",
            "params": {
                "threshold": 0.4
            }
        }
    },
    "transform": {
        "script": {
            "source": "HashMap result = new HashMap(); result.result = ctx.payload.aggregations.metricAgg.value; return result;",
            "lang": "painless",
            "params": {
                "threshold": 0.4
            }
        }
    },
    "actions": {
        "webhook_1": {
            "webhook": {
                "scheme": "http",
                "host": "api.platformdxc-d0.com",
                "port": 443,
                "method": "post",
                "path": "eve3-dev/dxc/events/R1/create",
                "params": {},
                "headers": {},
                "auth": {
                    "basic": {
                        "username": "useresam1",
                        "password": "::es_redacted::"
                    }
                },
                "body": "{\r\n \"EventList\": [\r\n {\r\n \"severity\": \"Minor\",\r\n \"incidentImpact\": 4,\r\n \"title\": \"IPopov2TestELK6\",\r\n \"node\": \"PdxcTestCI-DoNotDelete-3\",\r\n \"eventsourcesendingserver\": \"testhostRD2ELK1\",\r\n \"eventsourcebacksyncserver\": \"\",\r\n \"eventsourceexternalid\": \"testhostRD2ELK1\",\r\n \"eventsourcecreatedtime\": \"2019-12-11-09-20-38\",\r\n \"category\": \"OOSS\",\r\n \"application\": \"APP_NAME_RD2ELK1\",\r\n \"incidentCategory\": \"SoftwaretestRD2ELK1\",\r\n \"incidentSubcategory\": \"ApplicationtestRD2ELK1\"\r\n }\r\n]\r\n}"
            }
        }
    },
    "metadata": {
        "name": "CPU_metric_40",
        "watcherui": {
            "trigger_interval_unit": "m",
            "agg_type": "min",
            "time_field": "@timestamp",
            "trigger_interval_size": 10,
            "term_size": 5,
            "time_window_unit": "m",
            "threshold_comparator": ">",
            "index": [
                "metricbeat-*"
            ],
            "time_window_size": 5,
            "threshold": 0.4,
            "agg_field": "system.process.cpu.total.pct"
        },
        "xpack": {
            "type": "threshold"
        }
    }
}

```

}

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [December 19, 2019, 9:10am UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536/5 "2019-12-19T09:10:06Z")

</div>

the output of the execute watch API or the watcher history which shows a failed run is needed as well in order to help debugging this. Thank you!

--Alex

---

<div class="post-metadata">

**Author:** ![ipopov2](https://avatars.discourse-cdn.com/v4/letter/i/d26b3c/32.png) [@ipopov2](https://discuss.elastic.co/u/ipopov2)\
**Post date:** [December 19, 2019, 9:48am UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536/6 "2019-12-19T09:48:32Z")

</div>

Hi Alex, please find below the output of the watcher execution:

---

<div class="post-metadata">

**Author:** ![ipopov2](https://avatars.discourse-cdn.com/v4/letter/i/d26b3c/32.png) [@ipopov2](https://discuss.elastic.co/u/ipopov2)\
**Post date:** [December 19, 2019, 9:51am UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536/7 "2019-12-19T09:51:01Z")

</div>

```
{
"_id": "78c7e199-2f05-4975-a907-a2f5e8408fb1_35622fa6-75e7-4809-93cf-81b9c5e858ae-2019-12-19T09:40:49.742522Z",
"watch_record": {
    "watch_id": "78c7e199-2f05-4975-a907-a2f5e8408fb1",
    "node": "AZL0AquYSbefLbK9gNpfXA",
    "state": "executed",
    "user": "elastic",
    "status": {
        "state": {
            "active": true,
            "timestamp": "2019-12-13T07:20:00.500Z"
        },
        "last_checked": "2019-12-19T09:40:49.742Z",
        "last_met_condition": "2019-12-19T09:40:49.742Z",
        "actions": {
            "webhook_1": {
                "ack": {
                    "timestamp": "2019-12-13T07:20:00.500Z",
                    "state": "awaits_successful_execution"
                }
            }
        },
        "execution_state": "executed",
        "version": 878
    },
    "trigger_event": {
        "type": "manual",
        "triggered_time": "2019-12-19T09:40:49.742Z",
        "manual": {
            "schedule": {
                "scheduled_time": "2019-12-19T09:40:49.742Z"
            }
        }
    },
    "input": {
        "search": {
            "request": {
                "search_type": "query_then_fetch",
                "indices": [
                    "metricbeat-*"
                ],
                "rest_total_hits_as_int": true,
                "body": {
                    "size": 0,
                    "query": {
                        "bool": {
                            "filter": {
                                "range": {
                                    "@timestamp": {
                                        "gte": "{{ctx.trigger.scheduled_time}}||-5m",
                                        "lte": "{{ctx.trigger.scheduled_time}}",
                                        "format": "strict_date_optional_time||epoch_millis"
                                    }
                                }
                            }
                        }
                    },
                    "aggs": {
                        "metricAgg": {
                            "min": {
                                "field": "system.process.cpu.total.pct"
                            }
                        }
                    }
                }
            }
        }
    },
    "condition": {
        "script": {
            "source": "if (ctx.payload.aggregations.metricAgg.value > params.threshold) { return true; } return false;",
            "lang": "painless",
            "params": {
                "threshold": 0.4
            }
        }
    },
    "metadata": {
        "name": "CPU_metric_40",
        "watcherui": {
            "trigger_interval_unit": "m",
            "agg_type": "min",
            "time_field": "@timestamp",
            "trigger_interval_size": 10,
            "term_size": 5,
            "time_window_unit": "m",
            "threshold_comparator": ">",
            "index": [
                "metricbeat-*"
            ],
            "time_window_size": 5,
            "threshold": 0.4,
            "agg_field": "system.process.cpu.total.pct"
        },
        "xpack": {
            "type": "threshold"
        }
    },
    "result": {
        "execution_time": "2019-12-19T09:40:49.742Z",
        "execution_duration": 2,
        "input": {
            "type": "simple",
            "status": "success",
            "payload": {
                "foo": "bar"
            }
        },
        "condition": {
            "type": "always",
            "status": "success",
            "met": true
        },
        "transform": {
            "type": "script",
            "status": "failure",
            "reason": "runtime error",
            "error": {
                "root_cause": [
                    {
                        "type": "script_exception",
                        "reason": "runtime error",
                        "script_stack": [
                            "result.result = ctx.payload.aggregations.metricAgg.value; ",
                            " ^---- HERE"
                        ],
                        "script": "HashMap result = new HashMap(); result.result = ctx.payload.aggregations.metricAgg.value; return result;",
                        "lang": "painless"
                    }
                ],
                "type": "script_exception",
                "reason": "runtime error",
                "script_stack": [
                    "result.result = ctx.payload.aggregations.metricAgg.value; ",
                    " ^---- HERE"
                ],
                "script": "HashMap result = new HashMap(); result.result = ctx.payload.aggregations.metricAgg.value; return result;",
                "lang": "painless",
                "caused_by": {
                    "type": "null_pointer_exception",
                    "reason": null
                }
            }
        },
        "actions": []
    },
    "messages": [
        "failed to execute watch transform"
    ]
}

```

}

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [December 19, 2019, 9:52am UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536/8 "2019-12-19T09:52:33Z")

</div>

this run does not represent a real execution it seems. the payload does not contain the search response and thus the error is completely different to what you mentioned earlier in the issue.

---

<div class="post-metadata">

**Author:** ![ipopov2](https://avatars.discourse-cdn.com/v4/letter/i/d26b3c/32.png) [@ipopov2](https://discuss.elastic.co/u/ipopov2)\
**Post date:** [December 19, 2019, 12:30pm UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536/9 "2019-12-19T12:30:59Z")

</div>

I executed it via the API using POST on RESTAPIURL/\_watcher/watch/78c7e199-2f05-4975-a907-a2f5e8408fb1/\_execute with payload:

```
{

```

"trigger\_data" : {  
"triggered\_time" : "now",  
"scheduled\_time" : "now"  
},  
"alternative\_input" : {  
"foo" : "bar"  
},  
"ignore\_condition" : true,  
"action\_modes" : {  
"my-action" : "force\_simulate"  
},  
"record\_execution" : true  
}

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [December 19, 2019, 1:18pm UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536/10 "2019-12-19T13:18:40Z")

</div>

so the problem with the alternative input is, that no search is executed. This will not help debugging the issue. May I ask why you chose to execute using the above command?

--Alex

---

<div class="post-metadata">

**Author:** ![ipopov2](https://avatars.discourse-cdn.com/v4/letter/i/d26b3c/32.png) [@ipopov2](https://discuss.elastic.co/u/ipopov2)\
**Post date:** [December 19, 2019, 1:44pm UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536/11 "2019-12-19T13:44:35Z")

</div>

There is no particular reason. Can you pls advise me on the proper way of execution of the watcher?  
Thanks in advance.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [December 20, 2019, 9:00am UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536/12 "2019-12-20T09:00:44Z")

</div>

do not specify any body, see [https://www.elastic.co/guide/en/elasticsearch/reference/7.5/watcher-api-execute-watch.html](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/watcher-api-execute-watch.html)

---

<div class="post-metadata">

**Author:** ![ipopov2](https://avatars.discourse-cdn.com/v4/letter/i/d26b3c/32.png) [@ipopov2](https://discuss.elastic.co/u/ipopov2)\
**Post date:** [December 20, 2019, 11:38am UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536/13 "2019-12-20T11:38:31Z")

</div>

```
Thank for the advice. Here is the result of the execution with empty body:

{
"_id": "78c7e199-2f05-4975-a907-a2f5e8408fb1_68c93eae-cae4-42c6-b9ea-1b2be23cc6ed-2019-12-20T11:36:26.248747Z",
"watch_record": {
    "watch_id": "78c7e199-2f05-4975-a907-a2f5e8408fb1",
    "node": "AZL0AquYSbefLbK9gNpfXA",
    "state": "execution_not_needed",
    "user": "elastic",
    "status": {
        "state": {
            "active": true,
            "timestamp": "2019-12-13T07:20:00.500Z"
        },
        "last_checked": "2019-12-20T11:36:26.248Z",
        "last_met_condition": "2019-12-19T09:40:49.742Z",
        "actions": {
            "webhook_1": {
                "ack": {
                    "timestamp": "2019-12-13T07:20:00.500Z",
                    "state": "awaits_successful_execution"
                }
            }
        },
        "execution_state": "execution_not_needed",
        "version": 1035
    },
    "trigger_event": {
        "type": "manual",
        "triggered_time": "2019-12-20T11:36:26.248Z",
        "manual": {
            "schedule": {
                "scheduled_time": "2019-12-20T11:36:26.248Z"
            }
        }
    },
    "input": {
        "search": {
            "request": {
                "search_type": "query_then_fetch",
                "indices": [
                    "metricbeat-*"
                ],
                "rest_total_hits_as_int": true,
                "body": {
                    "size": 0,
                    "query": {
                        "bool": {
                            "filter": {
                                "range": {
                                    "@timestamp": {
                                        "gte": "{{ctx.trigger.scheduled_time}}||-5m",
                                        "lte": "{{ctx.trigger.scheduled_time}}",
                                        "format": "strict_date_optional_time||epoch_millis"
                                    }
                                }
                            }
                        }
                    },
                    "aggs": {
                        "metricAgg": {
                            "min": {
                                "field": "system.process.cpu.total.pct"
                            }
                        }
                    }
                }
            }
        }
    },
    "condition": {
        "script": {
            "source": "if (ctx.payload.aggregations.metricAgg.value > params.threshold) { return true; } return false;",
            "lang": "painless",
            "params": {
                "threshold": 0.4
            }
        }
    },
    "metadata": {
        "name": "CPU_metric_40",
        "watcherui": {
            "trigger_interval_unit": "m",
            "agg_type": "min",
            "time_field": "@timestamp",
            "trigger_interval_size": 10,
            "term_size": 5,
            "time_window_unit": "m",
            "threshold_comparator": ">",
            "index": [
                "metricbeat-*"
            ],
            "time_window_size": 5,
            "threshold": 0.4,
            "agg_field": "system.process.cpu.total.pct"
        },
        "xpack": {
            "type": "threshold"
        }
    },
    "result": {
        "execution_time": "2019-12-20T11:36:26.248Z",
        "execution_duration": 5,
        "input": {
            "type": "search",
            "status": "success",
            "payload": {
                "_shards": {
                    "total": 2,
                    "failed": 0,
                    "successful": 2,
                    "skipped": 0
                },
                "hits": {
                    "hits": [],
                    "total": 1648,
                    "max_score": null
                },
                "took": 4,
                "timed_out": false,
                "aggregations": {
                    "metricAgg": {
                        "value": 0.0
                    }
                }
            },
            "search": {
                "request": {
                    "search_type": "query_then_fetch",
                    "indices": [
                        "metricbeat-*"
                    ],
                    "rest_total_hits_as_int": true,
                    "body": {
                        "size": 0,
                        "query": {
                            "bool": {
                                "filter": {
                                    "range": {
                                        "@timestamp": {
                                            "gte": "2019-12-20T11:36:26.248732Z||-5m",
                                            "lte": "2019-12-20T11:36:26.248732Z",
                                            "format": "strict_date_optional_time||epoch_millis"
                                        }
                                    }
                                }
                            }
                        },
                        "aggs": {
                            "metricAgg": {
                                "min": {
                                    "field": "system.process.cpu.total.pct"
                                }
                            }
                        }
                    }
                }
            }
        },
        "condition": {
            "type": "script",
            "status": "success",
            "met": false
        },
        "actions": []
    },
    "messages": []
}

```

}

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [December 20, 2019, 12:39pm UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536/14 "2019-12-20T12:39:39Z")

</div>

the condition is not triggerde so we cannot check this further.

Is there any chance you could look at the watch history for this watch and see if you find a watch run where `met: true` was set?

You can search in the watcher history like this for runs of a certain watch

```auto
GET .watcher-history-*/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "watch_id": "my_watch"
          }
        }
      ]
    }
  },
  "sort": [
    {
      "trigger_event.triggered_time": {
        "order": "desc"
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![ipopov2](https://avatars.discourse-cdn.com/v4/letter/i/d26b3c/32.png) [@ipopov2](https://discuss.elastic.co/u/ipopov2)\
**Post date:** [December 20, 2019, 2:02pm UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536/15 "2019-12-20T14:02:16Z")

</div>

Just created new watcher with the following configuration:

```
{
"found": true,
"_id": "341dc68d-6dd6-4697-9680-1acdc666b128",
"_version": 9,
"_seq_no": 244950,
"_primary_term": 9,
"status": {
    "state": {
        "active": true,
        "timestamp": "2019-12-20T13:28:11.518Z"
    },
    "last_checked": "2019-12-20T13:36:12.002Z",
    "last_met_condition": "2019-12-20T13:36:12.002Z",
    "actions": {
        "webhook_1": {
            "ack": {
                "timestamp": "2019-12-20T13:29:11.717Z",
                "state": "ackable"
            },
            "last_execution": {
                "timestamp": "2019-12-20T13:36:12.002Z",
                "successful": true
            },
            "last_successful_execution": {
                "timestamp": "2019-12-20T13:36:12.002Z",
                "successful": true
            }
        }
    },
    "execution_state": "executed",
    "version": 9
},
"watch": {
    "trigger": {
        "schedule": {
            "interval": "1m"
        }
    },
    "input": {
        "search": {
            "request": {
                "search_type": "query_then_fetch",
                "indices": [
                    "metricbeat-*"
                ],
                "rest_total_hits_as_int": true,
                "body": {
                    "size": 0,
                    "query": {
                        "bool": {
                            "filter": {
                                "range": {
                                    "@timestamp": {
                                        "gte": "{{ctx.trigger.scheduled_time}}||-5m",
                                        "lte": "{{ctx.trigger.scheduled_time}}",
                                        "format": "strict_date_optional_time||epoch_millis"
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    },
    "condition": {
        "script": {
            "source": "if (ctx.payload.hits.total > params.threshold) { return true; } return false;",
            "lang": "painless",
            "params": {
                "threshold": 100
            }
        }
    },
    "transform": {
        "script": {
            "source": "HashMap result = new HashMap(); result.result = ctx.payload.hits.total; return result;",
            "lang": "painless",
            "params": {
                "threshold": 100
            }
        }
    },
    "actions": {
        "webhook_1": {
            "webhook": {
                "scheme": "http",
                "host": "api.platformdxc-d0.com/eve3-dev/dxc/events/R1/create",
                "port": 443,
                "method": "post",
                "params": {},
                "headers": {},
                "auth": {
                    "basic": {
                        "username": "useresam1",
                        "password": "::es_redacted::"
                    }
                },
                "body": "{\r\n \"EventList\": [\r\n {\r\n \"severity\": \"Minor\",\r\n \"incidentImpact\": 4,\r\n \"title\": \"IPopov2TestELK11\",\r\n \"node\": \"PdxcTestCI-DoNotDelete-3\",\r\n \"eventsourcesendingserver\": \"testhostRD2ELK1\",\r\n \"eventsourcebacksyncserver\": \"\",\r\n \"eventsourceexternalid\": \"testhostRD2ELK1\",\r\n \"eventsourcecreatedtime\": \"2019-12-11-09-20-38\",\r\n \"category\": \"OOSS\",\r\n \"application\": \"APP_NAME_RD2ELK1\",\r\n \"incidentCategory\": \"SoftwaretestRD2ELK1\",\r\n \"incidentSubcategory\": \"ApplicationtestRD2ELK1\"\r\n }\r\n]\r\n}\r\n"
            }
        }
    },
    "metadata": {
        "name": "Dummy_2",
        "watcherui": {
            "trigger_interval_unit": "m",
            "agg_type": "count",
            "time_field": "@timestamp",
            "trigger_interval_size": 1,
            "term_size": 5,
            "time_window_unit": "m",
            "threshold_comparator": ">",
            "index": [
                "metricbeat-*"
            ],
            "time_window_size": 5,
            "threshold": 100
        },
        "xpack": {
            "type": "threshold"
        }
    }
}

```

}

Splitting the details in two sections.

---

<div class="post-metadata">

**Author:** ![ipopov2](https://avatars.discourse-cdn.com/v4/letter/i/d26b3c/32.png) [@ipopov2](https://discuss.elastic.co/u/ipopov2)\
**Post date:** [December 20, 2019, 2:03pm UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536/16 "2019-12-20T14:03:04Z")

</div>

```
After executing it I got the following result:

{
"_id": "341dc68d-6dd6-4697-9680-1acdc666b128_56d72d12-979d-455c-a48b-8b40c359b44e-2019-12-20T13:47:04.371389Z",
"watch_record": {
    "watch_id": "341dc68d-6dd6-4697-9680-1acdc666b128",
    "node": "AZL0AquYSbefLbK9gNpfXA",
    "state": "executed",
    "user": "elastic",
    "status": {
        "state": {
            "active": true,
            "timestamp": "2019-12-20T13:28:11.518Z"
        },
        "last_checked": "2019-12-20T13:47:04.371Z",
        "last_met_condition": "2019-12-20T13:47:04.371Z",
        "actions": {
            "webhook_1": {
                "ack": {
                    "timestamp": "2019-12-20T13:29:11.717Z",
                    "state": "ackable"
                },
                "last_execution": {
                    "timestamp": "2019-12-20T13:47:04.371Z",
                    "successful": true
                },
                "last_successful_execution": {
                    "timestamp": "2019-12-20T13:47:04.371Z",
                    "successful": true
                }
            }
        },
        "execution_state": "executed",
        "version": 19
    },
    "trigger_event": {
        "type": "manual",
        "triggered_time": "2019-12-20T13:47:04.371Z",
        "manual": {
            "schedule": {
                "scheduled_time": "2019-12-20T13:47:04.371Z"
            }
        }
    },
    "input": {
        "search": {
            "request": {
                "search_type": "query_then_fetch",
                "indices": [
                    "metricbeat-*"
                ],
                "rest_total_hits_as_int": true,
                "body": {
                    "size": 0,
                    "query": {
                        "bool": {
                            "filter": {
                                "range": {
                                    "@timestamp": {
                                        "gte": "{{ctx.trigger.scheduled_time}}||-5m",
                                        "lte": "{{ctx.trigger.scheduled_time}}",
                                        "format": "strict_date_optional_time||epoch_millis"
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    },
    "condition": {
        "script": {
            "source": "if (ctx.payload.hits.total > params.threshold) { return true; } return false;",
            "lang": "painless",
            "params": {
                "threshold": 100
            }
        }
    },
    "metadata": {
        "name": "Dummy_2",
        "watcherui": {
            "trigger_interval_unit": "m",
            "agg_type": "count",
            "time_field": "@timestamp",
            "trigger_interval_size": 1,
            "term_size": 5,
            "time_window_unit": "m",
            "threshold_comparator": ">",
            "index": [
                "metricbeat-*"
            ],
            "time_window_size": 5,
            "threshold": 100
        },
        "xpack": {
            "type": "threshold"
        }
    },
    "result": {
        "execution_time": "2019-12-20T13:47:04.371Z",
        "execution_duration": 47,
        "input": {
            "type": "search",
            "status": "success",
            "payload": {
                "_shards": {
                    "total": 2,
                    "failed": 0,
                    "successful": 2,
                    "skipped": 0
                },
                "hits": {
                    "hits": [],
                    "total": 1633,
                    "max_score": null
                },
                "took": 3,
                "timed_out": false
            },
            "search": {
                "request": {
                    "search_type": "query_then_fetch",
                    "indices": [
                        "metricbeat-*"
                    ],
                    "rest_total_hits_as_int": true,
                    "body": {
                        "size": 0,
                        "query": {
                            "bool": {
                                "filter": {
                                    "range": {
                                        "@timestamp": {
                                            "gte": "2019-12-20T13:47:04.371374Z||-5m",
                                            "lte": "2019-12-20T13:47:04.371374Z",
                                            "format": "strict_date_optional_time||epoch_millis"
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        },
        "condition": {
            "type": "script",
            "status": "success",
            "met": true
        },
        "transform": {
            "type": "script",
            "status": "success",
            "payload": {
                "result": 1633
            }
        },
        "actions": [
            {
                "id": "webhook_1",
                "type": "webhook",
                "status": "success",
                "webhook": {
                    "request": {
                        "host": "api.platformdxc-d0.com/eve3-dev/dxc/events/R1/create",
                        "port": 443,
                        "scheme": "http",
                        "method": "post",
                        "auth": {
                            "basic": {
                                "username": "useresam1",
                                "password": "::es_redacted::"
                            }
                        },
                        "body": "{\r\n \"EventList\": [\r\n {\r\n \"severity\": \"Minor\",\r\n \"incidentImpact\": 4,\r\n \"title\": \"IPopov2TestELK11\",\r\n \"node\": \"PdxcTestCI-DoNotDelete-3\",\r\n \"eventsourcesendingserver\": \"testhostRD2ELK1\",\r\n \"eventsourcebacksyncserver\": \"\",\r\n \"eventsourceexternalid\": \"testhostRD2ELK1\",\r\n \"eventsourcecreatedtime\": \"2019-12-11-09-20-38\",\r\n \"category\": \"OOSS\",\r\n \"application\": \"APP_NAME_RD2ELK1\",\r\n \"incidentCategory\": \"SoftwaretestRD2ELK1\",\r\n \"incidentSubcategory\": \"ApplicationtestRD2ELK1\"\r\n }\r\n]\r\n}\r\n"
                    },
                    "response": {
                        "status": 307,
                        "headers": {
                            "x-amz-cf-pop": [
                                "FRA6-C1"
                            ],
                            "date": [
                                "Fri, 20 Dec 2019 13:47:04 GMT"
                            ],
                            "server": [
                                "CloudFront"
                            ],
                            "content-length": [
                                "185"
                            ],
                            "connection": [
                                "keep-alive"
                            ],
                            "content-type": [
                                "text/html"
                            ],
                            "location": [
                                "https://api.platformdxc-d0.com/eve3-dev/dxc/events/R1/create:443"
                            ],
                            "x-cache": [
                                "Redirect from cloudfront"
                            ],
                            "x-amz-cf-id": [
                                "Ud4mdqWWfpQGE5jAY7oES69PVttlEDGakxoFCqRUeJ_T_vc2QD5Kew=="
                            ],
                            "via": [
                                "1.1 89c822bb1ce1445a7be6d1057088cfbf.cloudfront.net (CloudFront)"
                            ]
                        },
                        "body": "<html>\r\n<head><title>307 Temporary Redirect</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>307 Temporary Redirect</h1></center>\r\n<hr><center>CloudFront</center>\r\n</body>\r\n</html>\r\n"
                    }
                }
            }
        ]
    },
    "messages": []
}

```

}

In the "response" section can be seen that for some reason Amazon Cloudfront is redirecting the request to somewhere...  
Have you witnessed such a behavior when Elastic watcher is using webhook, trying to access REST API in AWS?  
Actually we got the same response when trying to reach the API endpoint on http, but not on https.

---

<div class="post-metadata">

**Author:** ![lcui\_dxc](https://avatars.discourse-cdn.com/v4/letter/l/8edcca/32.png) [@lcui\_dxc](https://discuss.elastic.co/u/lcui_dxc)\
**Post date:** [December 20, 2019, 3:59pm UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536/17 "2019-12-20T15:59:24Z")

</div>

Just checked with ServiceNow admins, they said that the redirection is because in our request we have "http" instead of "https", it got redirected. https is not only a different port, it's also a different protocol than http. So we may have to find out why it keeps getting http instead of https...

Using POSTMAN outside ELK all our requests were sent to Event API on AWS and the events were and incidents on SNOW were generated. Not sure why inside ELK, the same process hits issues...

Thanks Iliya and Alex for all your help and we really need to get to the bottom of this ASAP.

---

<div class="post-metadata">

**Author:** ![HowardtheDuck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/howardtheduck/32/43870_2.png) [@HowardtheDuck](https://discuss.elastic.co/u/HowardtheDuck)\
**Post date:** [December 20, 2019, 5:52pm UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536/18 "2019-12-20T17:52:21Z")

</div>

Would it be possible to share your elasticsearch.yml configuration settings as well Li & Iliya?

---

<div class="post-metadata">

**Author:** ![lcui\_dxc](https://avatars.discourse-cdn.com/v4/letter/l/8edcca/32.png) [@lcui\_dxc](https://discuss.elastic.co/u/lcui_dxc)\
**Post date:** [December 20, 2019, 6:10pm UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536/19 "2019-12-20T18:10:07Z")

</div>

Sure. here is the yml for ES datanodes (we only have one data node):

# ======================== Elasticsearch Configuration =========================

# 

# ---------------------------------- Cluster -----------------------------------

# 

# Use a descriptive name for your cluster:

# 

cluster.name: rd-azure-es

# 

# ------------------------------------ Node ------------------------------------

# 

# Use a descriptive name for the node:

# 

node.name: rd-azure-data01

# 

# Add custom attributes to the node:

# 

node.master: false  
node.data: true  
node.ingest: true

# 

#node.attr.rack: r1

# 

# ----------------------------------- Paths ------------------------------------

# 

path.data: /data001/elkstack/elasticsearch/data

# 

# Path to log files:

# 

path.logs: /data001/elkstack/elasticsearch/logs

# 

# 

# ---------------------------------- Network -----------------------------------

# 

# Set the bind address to a specific IP (IPv4 or IPv6):

# 

network.host: x.x.x.5

# 

# Set a custom port for HTTP:

# 

http.port: 9200

# 

# For more information, consult the network module documentation.

network.publish\_host: x.x.x.5  
transport.host: x.x.x.5  
transport.port: 9300

# 

# --------------------------------- Discovery ----------------------------------

# 

# Pass an initial list of hosts to perform discovery when this node is started:

# The default list of hosts is ["127.0.0.1", "[::1]"]

#discovery.zen.ping.unicast.hosts: ["x.x.x.4:9300","x.x.x.5:9300","x.x.x.10:9300"]  
discovery.seed\_hosts: ["x.x.x.4:9300"]

# 

# ---------------------------------- Various -----------------------------------

# 

# Require explicit names when deleting indices:

# 

# #action.destructive\_requires\_name: true xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification\_mode: certificate xpack.security.transport.ssl.keystore.path: /data001/elkstack/elasticsearch/config/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: /data001/elkstack/elasticsearch/config/elastic-certificates.p12 xpack.http.ssl.certificate: none xpack.monitoring.enabled: true xpack.monitoring.collection.enabled: true xpack.monitoring.collection.interval: 60s xpack.monitoring.collection.cluster.stats.timeout: 60s xpack.monitoring.history.duration: 90d xpack.watcher.enabled: true xpack.notification.email.account: sendgrid\_account: profile: sendgrid smtp: auth: true starttls.enable: false host: [smtp.sendgrid.net](http://smtp.sendgrid.net) port: 587 user: apikey

here is the yml for master node:

# ======================== Elasticsearch Configuration =========================

# ---------------------------------- Cluster -----------------------------------

# 

# Use a descriptive name for your cluster:

# 

cluster.name: rd-azure-es

# 

# ------------------------------------ Node ------------------------------------

# 

# Use a descriptive name for the node:

# 

node.name: rd-azure-master01  
node.master: true  
node.data: false  
node.ingest: true

# 

# Add custom attributes to the node:

# 

#node.attr.rack: r1

# 

# ----------------------------------- Paths ------------------------------------

# 

# 

path.data: /data001/elkstack/elasticsearch/data

# 

# Path to log files:

# 

path.logs: /data001/elkstack/elasticsearch/logs

# 

# ----------------------------------- Memory -----------------------------------

# 

# Lock the memory on startup:

# 

#bootstrap.memory\_lock: true

# 

# 

# ---------------------------------- Network -----------------------------------

# 

# Set the bind address to a specific IP (IPv4 or IPv6):

# 

network.host: y.y.y.4

# 

# Set a custom port for HTTP:

# 

http.port: 9200

# 

network.publish\_host: y.y.y.4  
transport.host: y.y.y.4  
transport.port: 9300

# 

# --------------------------------- Discovery ----------------------------------

# 

# Pass an initial list of hosts to perform discovery when this node is started:

# The default list of hosts is ["127.0.0.1", "[::1]"]

# 

#discovery.seed\_hosts: ["y.y.y.4","y.y.y.5","y.y.y.10"]  
#discovery.zen.ping.unicast.hosts: ["y.y.y.4:9300","y.y.y.5:9300","y.y.y.10:9300"]  
discovery.seed\_hosts: ["y.y.y.4:9300"]

# Bootstrap the cluster using an initial set of master-eligible nodes:

# 

cluster.initial\_master\_nodes: ["rd-azure-master01"]

# 

# For more information, consult the discovery and cluster formation module documentation.

# 

# ---------------------------------- Gateway -----------------------------------

# 

# Block initial recovery after a full cluster restart until N nodes are started:

# 

#gateway.recover\_after\_nodes: 3

# 

# For more information, consult the gateway module documentation.

# 

# ---------------------------------- Various -----------------------------------

# 

# Require explicit names when deleting indices:

# 

# #action.destructive\_requires\_name: true xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification\_mode: certificate xpack.security.transport.ssl.keystore.path: /data001/elkstack/elasticsearch/config/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: /data001/elkstack/elasticsearch/config/elastic-certificates.p12 xpack.http.ssl.certificate: none xpack.monitoring.enabled: true xpack.monitoring.collection.enabled: true xpack.monitoring.collection.interval: 60s xpack.monitoring.collection.cluster.stats.timeout: 60s xpack.monitoring.history.duration: 90d xpack.watcher.enabled: true xpack.notification.email.account: sendgrid\_account: profile: sendgrid smtp: auth: true starttls.enable: false host: [smtp.sendgrid.net](http://smtp.sendgrid.net) port: 587 user: apikey

Please let us know what the next step would be, thanks alot

Li

---

<div class="post-metadata">

**Author:** ![lcui\_dxc](https://avatars.discourse-cdn.com/v4/letter/l/8edcca/32.png) [@lcui\_dxc](https://discuss.elastic.co/u/lcui_dxc)\
**Post date:** [December 24, 2019, 4:16am UTC](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536/20 "2019-12-24T04:16:32Z")

</div>

Any updates, please?

[Next page](https://discuss.elastic.co/t/issue-with-integrating-watcher-webhook-with-servicenow/211536.md?page=2)