Watcher 2.3 hipchat action not working

Elastic Stack Elasticsearch
1

Hi,

I've been trying to set up a basic watcher, to send alerts to HipChat, on my v2.3 Elasticsearch cluster. However looking at the watch history tells me it has been consistently failing throwing 400-Bad request errors, as follows:
"actions" : [ {
"id" : "notify-hipchat",
"type" : "hipchat",
"status" : "failure",
"hipchat" : {
"account" : "notify-monitoring",
"sent_messages" : [ {
"status" : "failure",
"reason" : "Bad Request",
"request" : {
"host" : "api.hipchat.com",
"port" : 443,
"scheme" : "https",
"method" : "post",
"path" : "/v2/room/Room Name/notification",
"headers" : {
"Content-Type" : "application/json",
"Authorization" : "Bearer ValidTokenId"
},
"body" : "{\n "message" : "Test ES Cluster status is Green",\n "message_format" : "text",\n "color" : "green"\n}"
},
"response" : {
"status" : 400,
"headers" : {
"content-length" : [ "0" ],
"connection" : [ "Close" ]
}
},
"room" : "Room Name",
"message" : {
"body" : "Cluster status is Green",
"format" : "text",
"color" : "green"
}
} ]

Using RestConsole, I've verified that the under-question HipChat API calls work otherwise, hence the token is valid.

Mentioned below are the watcher installation details:
{"version":{"number":"2.3.0","build_hash":"0bd7e1459f2eee2318205f2a168bf2d9c6a8dac3","build_timestamp":"2016-03-29T19:51:42Z","build_snapshot":false},"tagline":"You Know, for Alerts & Automation"}

Any help would be deeply appreciated as I am blocked now. Thank you in advance.

2

Hey,

the returned status is 400. So there is some client side problem. Have you tried a room without a space in it or tried to escape it, using a plus sign or %20

--Alex

3

also can you show your full watch?

4

Hey Alex, I did try making the same request (from the above error logs), via Rest Console & it worked fine, so not sure if it should be related to having space in the room name.

Here is the watcher I am trying to create:
curl -XPUT '<ES_URL>/_watcher/watch/cluster_health_watch' -d '{
"trigger" : {
"schedule" : { "interval" : "10s" }
},
"input" : {
"http" : {
"request" : {
"host" : "localhost",
"port" : 9200,
"path" : "/_cluster/health"
}
}
},
"condition" : {
"compare" : {
"ctx.payload.status" : { "eq" : "green" }
}
},
"actions" : {
"notify-hipchat" : {
"hipchat" : {
"account" : "notify-monitoring",
"message" :{
"body": "Test ES Cluster status is Green",
"format": "text",
"color": "green"
}
}
}
}
}'

5

can you also share your account setup (without the tokens of course)?

6

Sure Alex. Thank you helping out. Here are the account set-up config :
watcher.actions.hipchat.service.account.notify-monitoring.profile: integration
watcher.actions.hipchat.service.account.notify-monitoring.auth_token: <auth_token>
watcher.actions.hipchat.service.account.notify-monitoring.room: Room Name

7

Hey,

again, have you tried to sent to a room, that does not contain a space or URL encoded the space? Your tool may do that automatically for you, while watcher does not.

--Alex

8

Hmm ok. Let me try that possibility as well. Will get back after verifying.

9

Phew! That worked. I used the roomId instead of the room name(which had spaces) & it worked fine. Thank you for the help, Alex :slight_smile:

10

Hey,

glad to see it working! Do you think we should improve our documentation any emphasize anything there to make sure noone else runs into this? happy to get any input here!

--Alex

11

Yeah Alex. We should update the documentation to explicitly mention that spaces in roomName won't work, as this is a common use case. Secondly, there is a way to debug the input received for the action (via the logging action), but I don't see anyway to troubleshoot the action. There are no failure logs generated in ES too.

12

Hey,

its intentional to not log anything into the files - imagine thousands of watches just doing logging, so you might miss errors on Elasticsearch itself. At some point one has to differentiate between system logs by Elasticsearch and application logs by watcher - which is why the letter part is put into the watch history.

You can also debug using the Execute Watch API and - as you did - using the watch History.

I'll try to update the documentation soon, thanks for your input!

--Alex