Issue with integrating watcher webhook with ServiceNow

Elastic Stack Elasticsearch
1

Hello there:

We face issue when integrating watcher webhook to integrate with ServiceNow.
Here is what we did and got…. If possible we’d prefer to set up a session so I may show you the details and learn from you.

We tested using Curl on linux commandline on kibana Server, we were able to create incidents on ServiceNow (working fine we can see the incidents there)
We tested using POSTMan, it works fine…

Here is how we set up the webhook...

watcher_issue
watcher_issue1173×653 36.1 KB

Here is how we set up the webhook.

{
"EventList": [
{
"severity": "Minor",
"incidentImpact": 4,
"title": "IPopov2TestELK5",
"node": "PdxcTestCI-DoNotDelete-3",
"eventsourcesendingserver": "testhostRD2ELK1",
"eventsourcebacksyncserver": "",
"eventsourceexternalid": "testhostRD2ELK1",
"eventsourcecreatedtime": "2019-12-11-09-20-38",
"category": "OOSS",
"application": "APP_NAME_RD2ELK1",
"incidentCategory": "SoftwaretestRD2ELK1",
"incidentSubcategory": "ApplicationtestRD2ELK1"
}
]
}

////////////////////////////

If we use port 443, we got “received [400] status code”

The kibana log generated following event when use port 443. The method we used was “POST” but it seems in the log it shows as “PUT”.

{"type":"response","@timestamp":"2019-12-11T19:49:08Z","tags":,"pid":93202,"method":"put","statusCode":200,"req":{"url":"/api/watcher/watch/execute","method":"put","headers":{"host":"104.46.57.143:5601","connection":"keep-alive","content-length":"1503","accept":"application/json, text/plain, /","origin":"http://logstash.143:5601","kbn-version":"7.3.2","user-agent":"Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://104.46.57.143:5601/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"24.14.147.207","userAgent":"24.14.147.207","referer":"http://104.46.57.143:5601/app/kibana"},"res":{"statusCode":200,"responseTime":63,"contentLength":9},"message":"PUT /api/watcher/watch/execute 200 63ms - 9.0B"}

If we use port 80, we got
"Sample request sent to api.xxxx-d0.com:80/eve3-dev/yyy/events/R1/create"

It seems the request was sent to ServiceNow, but it didn't show up on ServiceNow. The ServiceNow Admin checked and said the request even didn't reach to servicenow.

The kibana log generated following event when use port 443. The method we used was “POST” but it seems in the log it shows as “PUT”. Both entries in the kibana log are same…

{"type":"response","@timestamp":"2019-12-11T19:49:30Z","tags":,"pid":93202,"method":"put","statusCode":200,"req":{"url":"/api/watcher/watch/execute","method":"put","headers":{"host":"104.46.57.143:5601","connection":"keep-alive","content-length":"1501","accept":"application/json, text/plain, /","origin":"http://logstash:5601","kbn-version":"7.3.2","user-agent":"Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://104.46.57.143:5601/app/kibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"24.14.147.207","userAgent":"24.14.147.207","referer":"http://104.46.57.143:5601/app/kibana"},"res":{"statusCode":200,"responseTime":30,"contentLength":9},"message":"PUT /api/watcher/watch/execute 200 30ms - 9.0B"}

Any help would be highly appreciated...

Thanks

Li

2

can you share the full watch and the output of the execute watch API in a gist?

3

For this we created the watch on Kibana as showing about.

4

Hi spinscale, Li, here are the watcher settings:

{
"found": true,
"_id": "78c7e199-2f05-4975-a907-a2f5e8408fb1",
"_version": 864,
"_seq_no": 229920,
"_primary_term": 9,
"status": {
    "state": {
        "active": true,
        "timestamp": "2019-12-13T07:20:00.500Z"
    },
    "last_checked": "2019-12-19T07:21:56.464Z",
    "actions": {
        "webhook_1": {
            "ack": {
                "timestamp": "2019-12-13T07:20:00.500Z",
                "state": "awaits_successful_execution"
            }
        }
    },
    "execution_state": "execution_not_needed",
    "version": 864
},
"watch": {
    "trigger": {
        "schedule": {
            "interval": "10m"
        }
    },
    "input": {
        "search": {
            "request": {
                "search_type": "query_then_fetch",
                "indices": [
                    "metricbeat-*"
                ],
                "rest_total_hits_as_int": true,
                "body": {
                    "size": 0,
                    "query": {
                        "bool": {
                            "filter": {
                                "range": {
                                    "@timestamp": {
                                        "gte": "{{ctx.trigger.scheduled_time}}||-5m",
                                        "lte": "{{ctx.trigger.scheduled_time}}",
                                        "format": "strict_date_optional_time||epoch_millis"
                                    }
                                }
                            }
                        }
                    },
                    "aggs": {
                        "metricAgg": {
                            "min": {
                                "field": "system.process.cpu.total.pct"
                            }
                        }
                    }
                }
            }
        }
    },
    "condition": {
        "script": {
            "source": "if (ctx.payload.aggregations.metricAgg.value > params.threshold) { return true; } return false;",
            "lang": "painless",
            "params": {
                "threshold": 0.4
            }
        }
    },
    "transform": {
        "script": {
            "source": "HashMap result = new HashMap(); result.result = ctx.payload.aggregations.metricAgg.value; return result;",
            "lang": "painless",
            "params": {
                "threshold": 0.4
            }
        }
    },
    "actions": {
        "webhook_1": {
            "webhook": {
                "scheme": "http",
                "host": "api.platformdxc-d0.com",
                "port": 443,
                "method": "post",
                "path": "eve3-dev/dxc/events/R1/create",
                "params": {},
                "headers": {},
                "auth": {
                    "basic": {
                        "username": "useresam1",
                        "password": "::es_redacted::"
                    }
                },
                "body": "{\r\n    \"EventList\": [\r\n        {\r\n            \"severity\": \"Minor\",\r\n            \"incidentImpact\": 4,\r\n            \"title\": \"IPopov2TestELK6\",\r\n            \"node\": \"PdxcTestCI-DoNotDelete-3\",\r\n            \"eventsourcesendingserver\": \"testhostRD2ELK1\",\r\n            \"eventsourcebacksyncserver\": \"\",\r\n            \"eventsourceexternalid\": \"testhostRD2ELK1\",\r\n            \"eventsourcecreatedtime\": \"2019-12-11-09-20-38\",\r\n            \"category\": \"OOSS\",\r\n            \"application\": \"APP_NAME_RD2ELK1\",\r\n            \"incidentCategory\": \"SoftwaretestRD2ELK1\",\r\n            \"incidentSubcategory\": \"ApplicationtestRD2ELK1\"\r\n        }\r\n    ]\r\n}"
            }
        }
    },
    "metadata": {
        "name": "CPU_metric_40",
        "watcherui": {
            "trigger_interval_unit": "m",
            "agg_type": "min",
            "time_field": "@timestamp",
            "trigger_interval_size": 10,
            "term_size": 5,
            "time_window_unit": "m",
            "threshold_comparator": ">",
            "index": [
                "metricbeat-*"
            ],
            "time_window_size": 5,
            "threshold": 0.4,
            "agg_field": "system.process.cpu.total.pct"
        },
        "xpack": {
            "type": "threshold"
        }
    }
}

}

5

the output of the execute watch API or the watcher history which shows a failed run is needed as well in order to help debugging this. Thank you!

--Alex

6

Hi Alex, please find below the output of the watcher execution:

7 
{
"_id": "78c7e199-2f05-4975-a907-a2f5e8408fb1_35622fa6-75e7-4809-93cf-81b9c5e858ae-2019-12-19T09:40:49.742522Z",
"watch_record": {
    "watch_id": "78c7e199-2f05-4975-a907-a2f5e8408fb1",
    "node": "AZL0AquYSbefLbK9gNpfXA",
    "state": "executed",
    "user": "elastic",
    "status": {
        "state": {
            "active": true,
            "timestamp": "2019-12-13T07:20:00.500Z"
        },
        "last_checked": "2019-12-19T09:40:49.742Z",
        "last_met_condition": "2019-12-19T09:40:49.742Z",
        "actions": {
            "webhook_1": {
                "ack": {
                    "timestamp": "2019-12-13T07:20:00.500Z",
                    "state": "awaits_successful_execution"
                }
            }
        },
        "execution_state": "executed",
        "version": 878
    },
    "trigger_event": {
        "type": "manual",
        "triggered_time": "2019-12-19T09:40:49.742Z",
        "manual": {
            "schedule": {
                "scheduled_time": "2019-12-19T09:40:49.742Z"
            }
        }
    },
    "input": {
        "search": {
            "request": {
                "search_type": "query_then_fetch",
                "indices": [
                    "metricbeat-*"
                ],
                "rest_total_hits_as_int": true,
                "body": {
                    "size": 0,
                    "query": {
                        "bool": {
                            "filter": {
                                "range": {
                                    "@timestamp": {
                                        "gte": "{{ctx.trigger.scheduled_time}}||-5m",
                                        "lte": "{{ctx.trigger.scheduled_time}}",
                                        "format": "strict_date_optional_time||epoch_millis"
                                    }
                                }
                            }
                        }
                    },
                    "aggs": {
                        "metricAgg": {
                            "min": {
                                "field": "system.process.cpu.total.pct"
                            }
                        }
                    }
                }
            }
        }
    },
    "condition": {
        "script": {
            "source": "if (ctx.payload.aggregations.metricAgg.value > params.threshold) { return true; } return false;",
            "lang": "painless",
            "params": {
                "threshold": 0.4
            }
        }
    },
    "metadata": {
        "name": "CPU_metric_40",
        "watcherui": {
            "trigger_interval_unit": "m",
            "agg_type": "min",
            "time_field": "@timestamp",
            "trigger_interval_size": 10,
            "term_size": 5,
            "time_window_unit": "m",
            "threshold_comparator": ">",
            "index": [
                "metricbeat-*"
            ],
            "time_window_size": 5,
            "threshold": 0.4,
            "agg_field": "system.process.cpu.total.pct"
        },
        "xpack": {
            "type": "threshold"
        }
    },
    "result": {
        "execution_time": "2019-12-19T09:40:49.742Z",
        "execution_duration": 2,
        "input": {
            "type": "simple",
            "status": "success",
            "payload": {
                "foo": "bar"
            }
        },
        "condition": {
            "type": "always",
            "status": "success",
            "met": true
        },
        "transform": {
            "type": "script",
            "status": "failure",
            "reason": "runtime error",
            "error": {
                "root_cause": [
                    {
                        "type": "script_exception",
                        "reason": "runtime error",
                        "script_stack": [
                            "result.result = ctx.payload.aggregations.metricAgg.value; ",
                            "                                        ^---- HERE"
                        ],
                        "script": "HashMap result = new HashMap(); result.result = ctx.payload.aggregations.metricAgg.value; return result;",
                        "lang": "painless"
                    }
                ],
                "type": "script_exception",
                "reason": "runtime error",
                "script_stack": [
                    "result.result = ctx.payload.aggregations.metricAgg.value; ",
                    "                                        ^---- HERE"
                ],
                "script": "HashMap result = new HashMap(); result.result = ctx.payload.aggregations.metricAgg.value; return result;",
                "lang": "painless",
                "caused_by": {
                    "type": "null_pointer_exception",
                    "reason": null
                }
            }
        },
        "actions": []
    },
    "messages": [
        "failed to execute watch transform"
    ]
}

}

8

this run does not represent a real execution it seems. the payload does not contain the search response and thus the error is completely different to what you mentioned earlier in the issue.

9

I executed it via the API using POST on RESTAPIURL/_watcher/watch/78c7e199-2f05-4975-a907-a2f5e8408fb1/_execute with payload:

{

"trigger_data" : {
"triggered_time" : "now",
"scheduled_time" : "now"
},
"alternative_input" : {
"foo" : "bar"
},
"ignore_condition" : true,
"action_modes" : {
"my-action" : "force_simulate"
},
"record_execution" : true
}

10

so the problem with the alternative input is, that no search is executed. This will not help debugging the issue. May I ask why you chose to execute using the above command?

--Alex

11

There is no particular reason. Can you pls advise me on the proper way of execution of the watcher?
Thanks in advance.

12

do not specify any body, see https://www.elastic.co/guide/en/elasticsearch/reference/7.5/watcher-api-execute-watch.html

13 
Thank for the advice. Here is the result of the execution with empty body:

{
"_id": "78c7e199-2f05-4975-a907-a2f5e8408fb1_68c93eae-cae4-42c6-b9ea-1b2be23cc6ed-2019-12-20T11:36:26.248747Z",
"watch_record": {
    "watch_id": "78c7e199-2f05-4975-a907-a2f5e8408fb1",
    "node": "AZL0AquYSbefLbK9gNpfXA",
    "state": "execution_not_needed",
    "user": "elastic",
    "status": {
        "state": {
            "active": true,
            "timestamp": "2019-12-13T07:20:00.500Z"
        },
        "last_checked": "2019-12-20T11:36:26.248Z",
        "last_met_condition": "2019-12-19T09:40:49.742Z",
        "actions": {
            "webhook_1": {
                "ack": {
                    "timestamp": "2019-12-13T07:20:00.500Z",
                    "state": "awaits_successful_execution"
                }
            }
        },
        "execution_state": "execution_not_needed",
        "version": 1035
    },
    "trigger_event": {
        "type": "manual",
        "triggered_time": "2019-12-20T11:36:26.248Z",
        "manual": {
            "schedule": {
                "scheduled_time": "2019-12-20T11:36:26.248Z"
            }
        }
    },
    "input": {
        "search": {
            "request": {
                "search_type": "query_then_fetch",
                "indices": [
                    "metricbeat-*"
                ],
                "rest_total_hits_as_int": true,
                "body": {
                    "size": 0,
                    "query": {
                        "bool": {
                            "filter": {
                                "range": {
                                    "@timestamp": {
                                        "gte": "{{ctx.trigger.scheduled_time}}||-5m",
                                        "lte": "{{ctx.trigger.scheduled_time}}",
                                        "format": "strict_date_optional_time||epoch_millis"
                                    }
                                }
                            }
                        }
                    },
                    "aggs": {
                        "metricAgg": {
                            "min": {
                                "field": "system.process.cpu.total.pct"
                            }
                        }
                    }
                }
            }
        }
    },
    "condition": {
        "script": {
            "source": "if (ctx.payload.aggregations.metricAgg.value > params.threshold) { return true; } return false;",
            "lang": "painless",
            "params": {
                "threshold": 0.4
            }
        }
    },
    "metadata": {
        "name": "CPU_metric_40",
        "watcherui": {
            "trigger_interval_unit": "m",
            "agg_type": "min",
            "time_field": "@timestamp",
            "trigger_interval_size": 10,
            "term_size": 5,
            "time_window_unit": "m",
            "threshold_comparator": ">",
            "index": [
                "metricbeat-*"
            ],
            "time_window_size": 5,
            "threshold": 0.4,
            "agg_field": "system.process.cpu.total.pct"
        },
        "xpack": {
            "type": "threshold"
        }
    },
    "result": {
        "execution_time": "2019-12-20T11:36:26.248Z",
        "execution_duration": 5,
        "input": {
            "type": "search",
            "status": "success",
            "payload": {
                "_shards": {
                    "total": 2,
                    "failed": 0,
                    "successful": 2,
                    "skipped": 0
                },
                "hits": {
                    "hits": [],
                    "total": 1648,
                    "max_score": null
                },
                "took": 4,
                "timed_out": false,
                "aggregations": {
                    "metricAgg": {
                        "value": 0.0
                    }
                }
            },
            "search": {
                "request": {
                    "search_type": "query_then_fetch",
                    "indices": [
                        "metricbeat-*"
                    ],
                    "rest_total_hits_as_int": true,
                    "body": {
                        "size": 0,
                        "query": {
                            "bool": {
                                "filter": {
                                    "range": {
                                        "@timestamp": {
                                            "gte": "2019-12-20T11:36:26.248732Z||-5m",
                                            "lte": "2019-12-20T11:36:26.248732Z",
                                            "format": "strict_date_optional_time||epoch_millis"
                                        }
                                    }
                                }
                            }
                        },
                        "aggs": {
                            "metricAgg": {
                                "min": {
                                    "field": "system.process.cpu.total.pct"
                                }
                            }
                        }
                    }
                }
            }
        },
        "condition": {
            "type": "script",
            "status": "success",
            "met": false
        },
        "actions": []
    },
    "messages": []
}

}

14

the condition is not triggerde so we cannot check this further.

Is there any chance you could look at the watch history for this watch and see if you find a watch run where met: true was set?

You can search in the watcher history like this for runs of a certain watch

GET .watcher-history-*/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "watch_id": "my_watch"
          }
        }
      ]
    }
  },
  "sort": [
    {
      "trigger_event.triggered_time": {
        "order": "desc"
      }
    }
  ]
}
15

Just created new watcher with the following configuration:

{
"found": true,
"_id": "341dc68d-6dd6-4697-9680-1acdc666b128",
"_version": 9,
"_seq_no": 244950,
"_primary_term": 9,
"status": {
    "state": {
        "active": true,
        "timestamp": "2019-12-20T13:28:11.518Z"
    },
    "last_checked": "2019-12-20T13:36:12.002Z",
    "last_met_condition": "2019-12-20T13:36:12.002Z",
    "actions": {
        "webhook_1": {
            "ack": {
                "timestamp": "2019-12-20T13:29:11.717Z",
                "state": "ackable"
            },
            "last_execution": {
                "timestamp": "2019-12-20T13:36:12.002Z",
                "successful": true
            },
            "last_successful_execution": {
                "timestamp": "2019-12-20T13:36:12.002Z",
                "successful": true
            }
        }
    },
    "execution_state": "executed",
    "version": 9
},
"watch": {
    "trigger": {
        "schedule": {
            "interval": "1m"
        }
    },
    "input": {
        "search": {
            "request": {
                "search_type": "query_then_fetch",
                "indices": [
                    "metricbeat-*"
                ],
                "rest_total_hits_as_int": true,
                "body": {
                    "size": 0,
                    "query": {
                        "bool": {
                            "filter": {
                                "range": {
                                    "@timestamp": {
                                        "gte": "{{ctx.trigger.scheduled_time}}||-5m",
                                        "lte": "{{ctx.trigger.scheduled_time}}",
                                        "format": "strict_date_optional_time||epoch_millis"
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    },
    "condition": {
        "script": {
            "source": "if (ctx.payload.hits.total > params.threshold) { return true; } return false;",
            "lang": "painless",
            "params": {
                "threshold": 100
            }
        }
    },
    "transform": {
        "script": {
            "source": "HashMap result = new HashMap(); result.result = ctx.payload.hits.total; return result;",
            "lang": "painless",
            "params": {
                "threshold": 100
            }
        }
    },
    "actions": {
        "webhook_1": {
            "webhook": {
                "scheme": "http",
                "host": "api.platformdxc-d0.com/eve3-dev/dxc/events/R1/create",
                "port": 443,
                "method": "post",
                "params": {},
                "headers": {},
                "auth": {
                    "basic": {
                        "username": "useresam1",
                        "password": "::es_redacted::"
                    }
                },
                "body": "{\r\n    \"EventList\": [\r\n        {\r\n            \"severity\": \"Minor\",\r\n            \"incidentImpact\": 4,\r\n            \"title\": \"IPopov2TestELK11\",\r\n            \"node\": \"PdxcTestCI-DoNotDelete-3\",\r\n            \"eventsourcesendingserver\": \"testhostRD2ELK1\",\r\n            \"eventsourcebacksyncserver\": \"\",\r\n            \"eventsourceexternalid\": \"testhostRD2ELK1\",\r\n            \"eventsourcecreatedtime\": \"2019-12-11-09-20-38\",\r\n            \"category\": \"OOSS\",\r\n            \"application\": \"APP_NAME_RD2ELK1\",\r\n            \"incidentCategory\": \"SoftwaretestRD2ELK1\",\r\n            \"incidentSubcategory\": \"ApplicationtestRD2ELK1\"\r\n        }\r\n    ]\r\n}\r\n"
            }
        }
    },
    "metadata": {
        "name": "Dummy_2",
        "watcherui": {
            "trigger_interval_unit": "m",
            "agg_type": "count",
            "time_field": "@timestamp",
            "trigger_interval_size": 1,
            "term_size": 5,
            "time_window_unit": "m",
            "threshold_comparator": ">",
            "index": [
                "metricbeat-*"
            ],
            "time_window_size": 5,
            "threshold": 100
        },
        "xpack": {
            "type": "threshold"
        }
    }
}

}

Splitting the details in two sections.

16 
After executing it I got the following result:

{
"_id": "341dc68d-6dd6-4697-9680-1acdc666b128_56d72d12-979d-455c-a48b-8b40c359b44e-2019-12-20T13:47:04.371389Z",
"watch_record": {
    "watch_id": "341dc68d-6dd6-4697-9680-1acdc666b128",
    "node": "AZL0AquYSbefLbK9gNpfXA",
    "state": "executed",
    "user": "elastic",
    "status": {
        "state": {
            "active": true,
            "timestamp": "2019-12-20T13:28:11.518Z"
        },
        "last_checked": "2019-12-20T13:47:04.371Z",
        "last_met_condition": "2019-12-20T13:47:04.371Z",
        "actions": {
            "webhook_1": {
                "ack": {
                    "timestamp": "2019-12-20T13:29:11.717Z",
                    "state": "ackable"
                },
                "last_execution": {
                    "timestamp": "2019-12-20T13:47:04.371Z",
                    "successful": true
                },
                "last_successful_execution": {
                    "timestamp": "2019-12-20T13:47:04.371Z",
                    "successful": true
                }
            }
        },
        "execution_state": "executed",
        "version": 19
    },
    "trigger_event": {
        "type": "manual",
        "triggered_time": "2019-12-20T13:47:04.371Z",
        "manual": {
            "schedule": {
                "scheduled_time": "2019-12-20T13:47:04.371Z"
            }
        }
    },
    "input": {
        "search": {
            "request": {
                "search_type": "query_then_fetch",
                "indices": [
                    "metricbeat-*"
                ],
                "rest_total_hits_as_int": true,
                "body": {
                    "size": 0,
                    "query": {
                        "bool": {
                            "filter": {
                                "range": {
                                    "@timestamp": {
                                        "gte": "{{ctx.trigger.scheduled_time}}||-5m",
                                        "lte": "{{ctx.trigger.scheduled_time}}",
                                        "format": "strict_date_optional_time||epoch_millis"
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    },
    "condition": {
        "script": {
            "source": "if (ctx.payload.hits.total > params.threshold) { return true; } return false;",
            "lang": "painless",
            "params": {
                "threshold": 100
            }
        }
    },
    "metadata": {
        "name": "Dummy_2",
        "watcherui": {
            "trigger_interval_unit": "m",
            "agg_type": "count",
            "time_field": "@timestamp",
            "trigger_interval_size": 1,
            "term_size": 5,
            "time_window_unit": "m",
            "threshold_comparator": ">",
            "index": [
                "metricbeat-*"
            ],
            "time_window_size": 5,
            "threshold": 100
        },
        "xpack": {
            "type": "threshold"
        }
    },
    "result": {
        "execution_time": "2019-12-20T13:47:04.371Z",
        "execution_duration": 47,
        "input": {
            "type": "search",
            "status": "success",
            "payload": {
                "_shards": {
                    "total": 2,
                    "failed": 0,
                    "successful": 2,
                    "skipped": 0
                },
                "hits": {
                    "hits": [],
                    "total": 1633,
                    "max_score": null
                },
                "took": 3,
                "timed_out": false
            },
            "search": {
                "request": {
                    "search_type": "query_then_fetch",
                    "indices": [
                        "metricbeat-*"
                    ],
                    "rest_total_hits_as_int": true,
                    "body": {
                        "size": 0,
                        "query": {
                            "bool": {
                                "filter": {
                                    "range": {
                                        "@timestamp": {
                                            "gte": "2019-12-20T13:47:04.371374Z||-5m",
                                            "lte": "2019-12-20T13:47:04.371374Z",
                                            "format": "strict_date_optional_time||epoch_millis"
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        },
        "condition": {
            "type": "script",
            "status": "success",
            "met": true
        },
        "transform": {
            "type": "script",
            "status": "success",
            "payload": {
                "result": 1633
            }
        },
        "actions": [
            {
                "id": "webhook_1",
                "type": "webhook",
                "status": "success",
                "webhook": {
                    "request": {
                        "host": "api.platformdxc-d0.com/eve3-dev/dxc/events/R1/create",
                        "port": 443,
                        "scheme": "http",
                        "method": "post",
                        "auth": {
                            "basic": {
                                "username": "useresam1",
                                "password": "::es_redacted::"
                            }
                        },
                        "body": "{\r\n    \"EventList\": [\r\n        {\r\n            \"severity\": \"Minor\",\r\n            \"incidentImpact\": 4,\r\n            \"title\": \"IPopov2TestELK11\",\r\n            \"node\": \"PdxcTestCI-DoNotDelete-3\",\r\n            \"eventsourcesendingserver\": \"testhostRD2ELK1\",\r\n            \"eventsourcebacksyncserver\": \"\",\r\n            \"eventsourceexternalid\": \"testhostRD2ELK1\",\r\n            \"eventsourcecreatedtime\": \"2019-12-11-09-20-38\",\r\n            \"category\": \"OOSS\",\r\n            \"application\": \"APP_NAME_RD2ELK1\",\r\n            \"incidentCategory\": \"SoftwaretestRD2ELK1\",\r\n            \"incidentSubcategory\": \"ApplicationtestRD2ELK1\"\r\n        }\r\n    ]\r\n}\r\n"
                    },
                    "response": {
                        "status": 307,
                        "headers": {
                            "x-amz-cf-pop": [
                                "FRA6-C1"
                            ],
                            "date": [
                                "Fri, 20 Dec 2019 13:47:04 GMT"
                            ],
                            "server": [
                                "CloudFront"
                            ],
                            "content-length": [
                                "185"
                            ],
                            "connection": [
                                "keep-alive"
                            ],
                            "content-type": [
                                "text/html"
                            ],
                            "location": [
                                "https://api.platformdxc-d0.com/eve3-dev/dxc/events/R1/create:443"
                            ],
                            "x-cache": [
                                "Redirect from cloudfront"
                            ],
                            "x-amz-cf-id": [
                                "Ud4mdqWWfpQGE5jAY7oES69PVttlEDGakxoFCqRUeJ_T_vc2QD5Kew=="
                            ],
                            "via": [
                                "1.1 89c822bb1ce1445a7be6d1057088cfbf.cloudfront.net (CloudFront)"
                            ]
                        },
                        "body": "<html>\r\n<head><title>307 Temporary Redirect</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>307 Temporary Redirect</h1></center>\r\n<hr><center>CloudFront</center>\r\n</body>\r\n</html>\r\n"
                    }
                }
            }
        ]
    },
    "messages": []
}

}

In the "response" section can be seen that for some reason Amazon Cloudfront is redirecting the request to somewhere...
Have you witnessed such a behavior when Elastic watcher is using webhook, trying to access REST API in AWS?
Actually we got the same response when trying to reach the API endpoint on http, but not on https.

17

Just checked with ServiceNow admins, they said that the redirection is because in our request we have "http" instead of "https", it got redirected. https is not only a different port, it's also a different protocol than http. So we may have to find out why it keeps getting http instead of https...

Using POSTMAN outside ELK all our requests were sent to Event API on AWS and the events were and incidents on SNOW were generated. Not sure why inside ELK, the same process hits issues...

Thanks Iliya and Alex for all your help and we really need to get to the bottom of this ASAP.

18

Would it be possible to share your elasticsearch.yml configuration settings as well Li & Iliya?

19

Sure. here is the yml for ES datanodes (we only have one data node):

======================== Elasticsearch Configuration =========================

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

cluster.name: rd-azure-es

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

node.name: rd-azure-data01

Add custom attributes to the node:

node.master: false
node.data: true
node.ingest: true

#node.attr.rack: r1

----------------------------------- Paths ------------------------------------

path.data: /data001/elkstack/elasticsearch/data

Path to log files:

path.logs: /data001/elkstack/elasticsearch/logs

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

network.host: x.x.x.5

Set a custom port for HTTP:

http.port: 9200

For more information, consult the network module documentation.

network.publish_host: x.x.x.5
transport.host: x.x.x.5
transport.port: 9300

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when this node is started:

The default list of hosts is ["127.0.0.1", "[::1]"]

#discovery.zen.ping.unicast.hosts: ["x.x.x.4:9300","x.x.x.5:9300","x.x.x.10:9300"]
discovery.seed_hosts: ["x.x.x.4:9300"]

---------------------------------- Various -----------------------------------

Require explicit names when deleting indices:

#action.destructive_requires_name: true
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /data001/elkstack/elasticsearch/config/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /data001/elkstack/elasticsearch/config/elastic-certificates.p12
xpack.http.ssl.certificate: none
xpack.monitoring.enabled: true
xpack.monitoring.collection.enabled: true
xpack.monitoring.collection.interval: 60s
xpack.monitoring.collection.cluster.stats.timeout: 60s
xpack.monitoring.history.duration: 90d
xpack.watcher.enabled: true
xpack.notification.email.account:
sendgrid_account:
profile: sendgrid
smtp:
auth: true
starttls.enable: false
host: smtp.sendgrid.net
port: 587
user: apikey

here is the yml for master node:

======================== Elasticsearch Configuration =========================

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

cluster.name: rd-azure-es

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

node.name: rd-azure-master01
node.master: true
node.data: false
node.ingest: true

Add custom attributes to the node:

#node.attr.rack: r1

----------------------------------- Paths ------------------------------------

path.data: /data001/elkstack/elasticsearch/data

Path to log files:

path.logs: /data001/elkstack/elasticsearch/logs

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

#bootstrap.memory_lock: true

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

network.host: y.y.y.4

Set a custom port for HTTP:

http.port: 9200

network.publish_host: y.y.y.4
transport.host: y.y.y.4
transport.port: 9300

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when this node is started:

The default list of hosts is ["127.0.0.1", "[::1]"]

#discovery.seed_hosts: ["y.y.y.4","y.y.y.5","y.y.y.10"]
#discovery.zen.ping.unicast.hosts: ["y.y.y.4:9300","y.y.y.5:9300","y.y.y.10:9300"]
discovery.seed_hosts: ["y.y.y.4:9300"]

Bootstrap the cluster using an initial set of master-eligible nodes:

cluster.initial_master_nodes: ["rd-azure-master01"]

For more information, consult the discovery and cluster formation module documentation.

---------------------------------- Gateway -----------------------------------

Block initial recovery after a full cluster restart until N nodes are started:

#gateway.recover_after_nodes: 3

For more information, consult the gateway module documentation.

---------------------------------- Various -----------------------------------

Require explicit names when deleting indices:

#action.destructive_requires_name: true
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /data001/elkstack/elasticsearch/config/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /data001/elkstack/elasticsearch/config/elastic-certificates.p12
xpack.http.ssl.certificate: none
xpack.monitoring.enabled: true
xpack.monitoring.collection.enabled: true
xpack.monitoring.collection.interval: 60s
xpack.monitoring.collection.cluster.stats.timeout: 60s
xpack.monitoring.history.duration: 90d
xpack.watcher.enabled: true
xpack.notification.email.account:
sendgrid_account:
profile: sendgrid
smtp:
auth: true
starttls.enable: false
host: smtp.sendgrid.net
port: 587
user: apikey

Please let us know what the next step would be, thanks alot

Li

20

Any updates, please?