java.net.ConnectException: Operation timed out when posting messages to slack incoming web hook

Elastic Stack Elasticsearch
1
  1. I have created new incoming web hook for my channel.
  2. Used incoming webhook in elasticsearch.yml file to send slack messages.

watcher.actions.slack.service:
account:
monitoring:
url: https://hooks.slack.com/services/xxx/xxx/xxxx
message_defaults:
from: Watcher

Created new watch to alert me when cluster status is yellow.

PUT /_watcher/watch/cluster_health_yellow_slack_watch
{
"trigger": {
"schedule": {
"interval": "30s"
}
},
"input" : {
"http" : {
"request" : {
"host" : "localhost",
"port" : 9200,
"path" : "/_cluster/health"
}
}
},

"condition" : {
"compare" : {
"ctx.payload.status" : { "eq" : "yellow" }
}
},
"actions": {
"send_trigger": {
"webhook": {
"method": "POST",
"scheme": "https",
"port": 443,
"host": "hooks.slack.com",
"path": "/services/xxx/xxx/xxx",
"body": "{ "text": "Cluster status is Yellow - 06 OCT 11:45 "}",
"headers": {
"Content-type": "application/json"
}
}
}
}
}

Below is the exception message I am seeing:

[2016-10-07 00:11:14,395][ERROR][watcher.actions.webhook ] [Trick Shot] failed to execute action [cluster_health_yellow_slack_watch/send_trigger]
java.net.ConnectException: Operation timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:275)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:371)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1147)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:998)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1282)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1257)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
at org.elasticsearch.watcher.support.http.HttpClient.doExecute(HttpClient.java:192)
at org.elasticsearch.watcher.support.http.HttpClient.execute(HttpClient.java:125)
at org.elasticsearch.watcher.actions.webhook.ExecutableWebhookAction.execute(ExecutableWebhookAction.java:57)
at org.elasticsearch.watcher.actions.ActionWrapper.execute(ActionWrapper.java:106)
at org.elasticsearch.watcher.execution.ExecutionService.executeInner(ExecutionService.java:388)
at org.elasticsearch.watcher.execution.ExecutionService.execute(ExecutionService.java:273)
at org.elasticsearch.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:438)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

NOTE: This issue is only when I am using my company network, Works fine on my home Wifi Connection.

2

Hey,

you already parttially answered your issue with your last sentence. The occuring timeout mentioned in the exception is very likely, because in your corporate network there are some filters in place, which dont allow the system, which watcher and elasticsearch is running on, to communicate with the internet.

Please sync with your system administrators what can be done about that. You might already get away by using a company proxy, but that depends on your company setup of course.

--Alex

3

Thank you Alex for the response.

Can we setup proxy in elasticsearch.yml to overcome this problem? I am using below properties
watcher.https.proxy.host: https://proxy.xxx.com
watcher.https.proxy.port: xxxx
watcher.http.proxy.host: http://proxy.xxx.com
watcher.http.proxy.port: xxxx

GET _nodes/settings is showing

{
"cluster_name": "elasticsearch",
"nodes": {
"JhbdQdjAR6O7dIPkE_eIPg": {
"name": "Soldier X",
"transport_address": "127.0.0.1:9300",
"host": "127.0.0.1",
"ip": "127.0.0.1",
"version": "2.3.4",
"build": "e455fd0",
"http_address": "127.0.0.1:9200",
"settings": {
"watcher": {
"actions": {
"slack": {
"service": {
"account": {
"monitoring": {
"message_defaults": {
"from": "Watcher"
},
"url": "https://hooks.slack.com/services/xxx/xxx/xxx"
}
}
}
}
},
"http": {
"proxy": {
"host": "http://proxy.xxx.com",
"port": "xxxx"
}
},
"https": {
"proxy": {
"host": "https://proxy.xxx.com",
"port": "xxxx"
}
}
},
"cluster": {
"name": "elasticsearch"
},
"path": {
"logs": "/Users/xxxxxx/Softwares/elasticsearch-2.3.4/logs",
"home": "/Users/xxxxx/Softwares/elasticsearch-2.3.4"
},
"shield": {
"enabled": "false"
},
"name": "Soldier X",
"client": {
"type": "node"
},
"config": {
"ignore_system_properties": "true"
},
"threadpool": {
"watcher": {
"queue_size": "1000",
"size": "40",
"type": "fixed"
}
},
"script": {
"inline": "true",
"indexed": "true"
}
}
}
}
}

I am doing it right?

4

Hey,

you only need watcher.http.proxy.* settings, see the docs.

Make sure you update that setting on all your nodes and restart them.

--Alex

5

Hey Alex,

How can I pass credentials for the proxy?

These two setting will work?
watcher.http.proxy.username: xxxx
watcher.http.proxy.password: xxxx

Please let me know

Prashanth

6

Hey,

passing credentials for the proxy is not supported at the moment, you can only pass credentials for the HTTP request as part of the watch.

We did not have a lot of requests regarding that, but it should not be too hard to implement I guess.

--Alex