Kibana Connections growing exponentially

[davidv (David)]

Hi,

Kibana 6.2.3
I have an ES cluster with a single Kibana instance, the connections sockets seem to not close properly and are constantly growing:

Checked logs - seems ok.
timeout configuration is as default.
The should be less than 100 connections.
Any idea where should I look into?

[jbudz (Jon Budzenski)]

Were you able to verify restarting the server brings it back down 0 (double checking UI bug vs connection leak)? Node.js should hard close any socket open more than 2 minutes so it's a tough one.

Do you have any load balancers or proxies in the setup?

[davidv (David)]

On restart it sets back to 0.
No load balancers or proxies.

[davidv (David)]

Hi @jbudz ,
No ideas?

[jbudz (Jon Budzenski)]

@davidv I did some digging, but nothing conclusive. This is a Kibana bug, and you're not the only one that has seen it. It appears to be fixed in newer versions, but I haven't been able to find the root cause, or where it was fixed. We have two major upgrades to our web server framework in between, and that's the most suspect area.

It may be worthwhile to check if this is the actual number of open connections, or a display bug. Does lsof -u <kibana_user> have a huge list of connections?

[davidv (David)]

@jbudz could you share the existing bugs?
lsof output is:

COMMAND  PID   USER   FD   TYPE  DEVICE   SIZE/OFF     NODE NAME
node    2631 kibana  cwd    DIR   253,0       4096        2 /
node    2631 kibana  rtd    DIR   253,0       4096        2 /
node    2631 kibana  txt    REG   253,0   30559647 24707090 /usr/share/kibana/node/bin/node
node    2631 kibana  mem    REG   253,0     161776 12968364 /lib64/ld-2.12.so
node    2631 kibana  mem    REG   253,0      23088 12968372 /lib64/libdl-2.12.so
node    2631 kibana  mem    REG   253,0    1930416 12968365 /lib64/libc-2.12.so
node    2631 kibana  mem    REG   253,0     146592 12968370 /lib64/libpthread-2.12.so
node    2631 kibana  mem    REG   253,0      47760 12968377 /lib64/librt-2.12.so
node    2631 kibana  mem    REG   253,0     600048 12968366 /lib64/libm-2.12.so
node    2631 kibana  DEL    REG   253,0            12968369 /lib64/libgcc_s-4.4.7-20120601.so.1;5c8a6fec
node    2631 kibana  DEL    REG   253,0            24512673 /usr/lib64/libstdc++.so.6.0.13
node    2631 kibana  mem    REG   253,0      66432 12967968 /lib64/libnss_files-2.12.so
node    2631 kibana    0r   CHR     1,3        0t0     4601 /dev/null
node    2631 kibana    1w   REG   253,0 1076205730 10584157 /var/log/kibana/kibana.stdout
node    2631 kibana    2w   REG   253,0  168323847 10584158 /var/log/kibana/kibana.stderr
node    2631 kibana    3r  FIFO     0,8        0t0    15388 pipe
node    2631 kibana    4w  FIFO     0,8        0t0    15388 pipe
node    2631 kibana    5u   REG     0,9          0     4597 [eventpoll]
node    2631 kibana    6r  FIFO     0,8        0t0    15389 pipe
node    2631 kibana    7w  FIFO     0,8        0t0    15389 pipe
node    2631 kibana    8u   REG     0,9          0     4597 [eventfd]
node    2631 kibana    9u  IPv4 3429020        0t0      TCP ElasticF:52154->ElasticF:wap-wsp (ESTABLISHED)
node    2631 kibana   10r   CHR     1,3        0t0     4601 /dev/null
node    2631 kibana   11u  IPv4 3429021        0t0      TCP ElasticF:52156->ElasticF:wap-wsp (ESTABLISHED)
node    2631 kibana   12u  IPv4   16917        0t0      TCP *:esmagent (LISTEN)
node    2631 kibana   13u  IPv4 3427395        0t0      TCP ElasticF:51980->ElasticF:wap-wsp (ESTABLISHED)
node    2631 kibana   14u  IPv4 3425786        0t0      TCP ElasticF:51812->ElasticF:wap-wsp (ESTABLISHED)
node    2631 kibana   15u  IPv4 3427396        0t0      TCP ElasticF:51982->ElasticF:wap-wsp (ESTABLISHED)
node    2631 kibana   16u  IPv4 3425891        0t0      TCP ElasticF:51828->ElasticF:wap-wsp (ESTABLISHED)
node    2631 kibana   17u  IPv4 3530303        0t0      TCP ElasticF:59306->ElasticF:wap-wsp (ESTABLISHED)
node    2631 kibana   18u  IPv4 3793098        0t0      TCP 10.65.151.75:esmagent-141.137.198.153:64441 (ESTABLISHED)
node    2631 kibana   19u  IPv4 3809909        0t0      TCP ElasticF:49064->ElasticF:wap-wsp (ESTABLISHED)
node    2631 kibana   20u  IPv4 3639689        0t0      TCP ElasticF:38646->ElasticF:wap-wsp (ESTABLISHED)
node    2631 kibana   21u  IPv4 3809910        0t0      TCP ElasticF:49066->ElasticF:wap-wsp (ESTABLISHED)
node    2631 kibana   22u  IPv4 3426189        0t0      TCP ElasticF:51868->ElasticF:wap-wsp (ESTABLISHED)
node    2631 kibana   24u  IPv4 3425981        0t0      TCP ElasticF:51840->ElasticF:wap-wsp (ESTABLISHED)
node    2631 kibana   26u  IPv4 3429024        0t0      TCP ElasticF:52160->ElasticF:wap-wsp (ESTABLISHED)
node    2631 kibana   27u  IPv4 3643016        0t0      TCP ElasticF:38868->ElasticF:wap-wsp (ESTABLISHED)
node    2631 kibana   28u  IPv4 3426195        0t0      TCP ElasticF:51874->ElasticF:wap-wsp (ESTABLISHED)
node    2631 kibana   29u  IPv4 3643017        0t0      TCP ElasticF:38870->ElasticF:wap-wsp (ESTABLISHED)
node    2631 kibana   31u  IPv4 3643019        0t0      TCP ElasticF:38872->ElasticF:wap-wsp (ESTABLISHED)
node    2631 kibana   32u  IPv4 3639704        0t0      TCP ElasticF:38652->ElasticF:wap-wsp (ESTABLISHED)