Reposting : Kibana 4.4.2 connects to Elastisearch 2.2.1 every 2-3 seconds


#1

Kibana keeps on creating new connection to Elastisearch every 2- 3 seconds and gets disconnected . Not sure which side disconnects but I see "Checking Elasticsearch version "message every 2-3 seconds.

{"type":"log","@timestamp":"2016-04-01T21:59:00+00:00","tags":["plugin","debug"],"pid":29894,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2016-04-01T21:59:02+00:00","tags":["plugin","debug"],"pid":29894,"message":"Checking Elasticsearch version"}
{"type":"ops","@timestamp":"2016-04-01T21:59:04+00:00","tags":[],"pid":29894,"os":{"load":[0.07373046875,0.0810546875,0.05322265625],"mem":{"total":8255295488,"free":3776933888},"uptime":1218427},"proc":{"uptime":11375.46,"mem":{"rss":293138432,"heapTotal":268594272,"heapUsed":262037528},"delay":0.4812660217285156},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"concurrents":{"5601":0},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 249.9MB uptime: 3:09:35 load: [0.07 0.08 0.05] delay: 0.481"}
{"type":"log","@timestamp":"2016-04-01T21:59:05+00:00","tags":["plugin","debug"],"pid":29894,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2016-04-01T21:59:07+00:00","tags":["plugin","debug"],"pid":29894,"message":"Checking Elasticsearch version"}
{"type":"ops","@timestamp":"2016-04-01T21:59:09+00:00","tags":[],"pid":29894,"os":{"load":[0.0673828125,0.07958984375,0.052734375],"mem":{"total":8255295488,"free":3776552960},"uptime":1218432},"proc":{"uptime":11380.462,"mem":{"rss":293511168,"heapTotal":268594272,"heapUsed":262418456},"delay":1.0139670372009277},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"concurrents":{"5601":0},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 250.3MB uptime: 3:09:40 load: [0.07 0.08 0.05] delay: 1.014"}
{"type":"log","@timestamp":"2016-04-01T21:59:10+00:00","tags":["plugin","debug"],"pid":29894,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2016-04-01T21:59:12+00:00","tags":["plugin","debug"],"pid":29894,"message":"Checking Elasticsearch version"}
{"type":"ops","@timestamp":"2016-04-01T21:59:14+00:00","tags":[],"pid":29894,"os":{"load":[0.0615234375,0.078125,0.05224609375],"mem":{"total":8255295488,"free":3776942080},"uptime":1218437},"proc":{"uptime":11385.465,"mem":{"rss":293195776,"heapTotal":268594272,"heapUsed":262141112},"delay":0.4728062152862549},"load":{"requests":{"5601":{"total":0,"disconnects":0,"statusCodes":{}}},"concurrents":{"5601":0},"responseTimes":{"5601":{"avg":null,"max":0}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 250.0MB uptime: 3:09:45 load: [0.06 0.08 0.05] delay: 0.473"}
{"type":"log","@timestamp":"2016-04-01T21:59:15+00:00","tags":["plugin","debug"],"pid":29894,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2016-04-01T21:59:17+00:00","tags":["plugin","debug"],"pid":29894,"message":"Checking Elasticsearch version"}

and
netstat -an | grep 9200 gives me around 70-92 tcp connections in TIME_WAIT state. Not sure why its doing this and would be great if you guys point me to a workaround or a fix.

tcp 0 0 127.0.0.1:53242 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:53222 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:53284 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:53237 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:53231 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:53229 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:53239 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:53254 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:53240 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:53280 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:53218 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:53270 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:53255 127.0.0.1:9200 TIME_WAIT
tcp 0 0 127.0.0.1:53213 127.0.0.1:9200 TIME_WAIT

Any suggestions on how to get rid of this is appreciated.


Kibana 4.4.2 connects to Elastisearch 2.2.1 every 2-3 seconds (Redhat 6)
(Tyler Smalley) #2

Kibana has a health check which is preformed against ElasticSearch to check it's status and version. There has been discussion on the frequency of this check.

I am not sure if that is related to the TIME_WAIT states. Do these continue to increase or are they eventually closed?


#3

Tyler,

There are always around 92 ports in TIME_WAIT state and I confirmed using wireshark that kibana opens and closes a tcp connection for every request that it sends to elasticsearch even when I'm not logged in to the kibana dashboard.

Thanks
Daya


(Tyler Smalley) #4

This appears to be normal, though I would like to see us use connection pooling to re-use a handful of connections instead of creating a new one each time. Are the number of TIME_WAIT connections posing a problem for you?


#5

No problems at all, now that I know that kibana opens a connection per REST request and it was not an error that was triggering the closing of the tcp connection.

Thanks
-Daya


(system) #6