Getsockopt: connection timed out

[Sam_WCL (Sam Anstee)]

Hello,
I am having an issue on a few boxes reporting to my elastic stack,
They look to be getting blocked/timing out with the below error in the metricbeat logs,

2018-08-07T12:33:49.810+0100 ERROR pipeline/output.go:74 Failed to connect: Get http://ELASTIC_DOMAIN:9200: dial tcp SERVER_IP:9200: getsockopt: connection timed out

I then ran a metricbeat test which returned the below,

elasticsearch: http://ELASTIC_DOMAIN:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: SERVER_IP
dial up... ERROR dial tcp SERVER_IP:9200: getsockopt: connection timed out

Can any one advise what might be the issue? I have 243 boxes that connect fine and im unsure of the amount that are having this issue. running a GET on the elasticsearch domain returns fine in 25-75MS.

Kind Regards,
Sam

[jsoriano (Jaime Soriano)]

Hi @Sam_WCL,

243 doesn't look like a big number for this but... could you be hitting some connection limit in your network or your hosts?
Could you try to connect from the affected hosts to the elasticsearch server or to other hosts using other tool like curl or telnet?

[Sam_WCL (Sam Anstee)]

Hello @jsoriano,
Thanks for getting back to me,
I ran the below command on a box that works and one that doesn't (both setup completely the same),

WORKING SERVER

curl -X GET ELASTIC_DOMAIN:9200
{
"name" : "PgUNPEj",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "J26G1zdaRzefxPUOTn6JRw",
"version" : {
"number" : "6.3.0",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "424e937",
"build_date" : "2018-06-11T23:38:03.357887Z",
"build_snapshot" : false,
"lucene_version" : "7.3.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

telnet SERVER_IP 9200
Trying SERVER_IP...
Connected to SERVER_IP.
Escape character is '^]'.

SERVER WITH ISSUES

curl -X GET ELASTIC_DOMAIN:9200
curl: (7) Failed connect to ELASTIC_DOMAIN:9200; Connection timed out

telnet SERVER_IP 9200
Trying SERVER_IP...
telnet: connect to address ELASTIC_IP: Connection timed out

[jsoriano (Jaime Soriano)]

This looks like a connectivity problem. Are both machines in the same network? is it possible that some firewall is blocking connections from some servers but not others?

[Sam_WCL (Sam Anstee)]

Hello @jsoriano,
Nothing looks to be getting blocked at Firewall level etc.
I think I will test building a replica elastic stack and change the host on a server with issues to point to the new one to see if that works.
I'll keep posting here with any updates.