Socket hang up error in kibana v 6.2.1

Elastic Stack Kibana
1

i am using Elasticsearch and kibana version 6.2.1.
When i enable SSL on kibana Webserver i am getting below error continuously in kibana logs.

{"type":"error","@timestamp":"2018-03-20T14:32:09Z","tags":["connection","client","error"],"pid":4784,"level":"error","error":{"message":"socket hang up","name":"Error","stack":"Error: socket hang up\n at TLSSocket.<anonymous> (_tls_wrap.js:847:25)\n at emitOne (events.js:101:20)\n at TLSSocket.emit (events.js:188:7)\n at _handle.close (net.js:497:12)\n at Socket.done (_tls_wrap.js:332:7)\n at Socket.g (events.js:292:16)\n at emitOne (events.js:101:20)\n at Socket.emit (events.js:188:7)\n at TCP._handle.close [as _onclose] (net.js:497:12)","code":"ECONNRESET"},"message":"socket hang up"}

2

An ECONNRESET usually means one side of the connection has suddenly closed it, e.g. because one end wasn't speaking SSL actually.

For better support, you would need to provide a bit more information. When do these logs appear? What happens when you try to access Kibana in your browser (or even better via curl)?

3

This error is logged continuously. When i login to kibana via browser it is working fine and i am able to see the data.

4

Have you enabled SSL also in your Elasticsearch cluster? What is your elasticsearch.url configuration in your kibana.yml?

5

yes i have enabled SSL on elasticsearch via readonlyREST.
here is my kibana config
server.port: 5601
server.host: xxxxxxxxxxxxxxxxx
server.maxPayloadBytes: 10485760
elasticsearch.url: https://xxxxxxxx:9200
elasticsearch.requestTimeout: 180000
kibana.index: .kibana
logging.dest: kibana.log
elasticsearch.username: "xxxxxx"
elasticsearch.password: "xxxxxxxxx"
elasticsearch.ssl.verificationMode: none
server.ssl.enabled: true
server.ssl.key: "keystore.key"
server.ssl.certificate: "certificate.crt"
server.ssl.keyPassphrase: xxxxx

6

Please help!

7

@vissu Two more suggestions what could be going wrong here. Since everything seems to work for you, that log will be cause by any other client out there, trying to connect to your Kibana instance.

  • Do you maybe have monitoring set up for that system, that is trying to load Kibana on that URL, but actually uses HTTP against that URL instead of HTTP (i.e. it's monitoring http://your-server.tld). That would explain the regularity of these dying connections.
  • Do you maybe have your Kibana instance exposed to the Internet? In this case it might be (malicious) bots, that are just crawling new URLs out there on all possible ports and try to connect. Usually if you put up a new server reachable from the internet, you can very quickly see, how it is spammed by connection attempts from bots.

To really see what is causing these, you should best use a tool like wireshark or tcpdump on the server, and see where these connections are coming from.

8

I'm also facing same problem.So giving answer as per my configuration

  • In kibana.yml , I've configured elasticsearch.url to https://hostname:port. The default value of xpack.monitoring.elasticsearch.url is elasticsearch.url so definitely its pointing to https.
  • No, our kibana application is deployed on our private VPN , its not exposed to the internet , so bot attack is also not possible.

Kibana continuously(after 30 sec) throwing this error , please suggest some solution.

9

Hi timroes,

I dont have any monitoring setup.
My kibana is not exposed to open internet, its under VPN.
i Checked TCP dump as well . i did not find any malicious bots.

10

If you checked the TCP dump, from what IP address did these requests came to the Kibana instance? Was it cleartext HTTP or was it encrypted HTTPS?

11

Hi timeroes,

The incoming connections were from LTM. The issue was with LTM configuration. Now i am not the error after changing LTM conf.
Thank you very much for your help!

12

Glad we could figure it out in the end :slight_smile:

Enjoy the day! :sunny:

1 Like
13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.