davidv
(David)
March 14, 2019, 8:31am
1
Hi,
Kibana 6.2.3
I have an ES cluster with a single Kibana instance, the connections sockets seem to not close properly and are constantly growing:
Checked logs - seems ok.
timeout configuration is as default.
The should be less than 100 connections.
Any idea where should I look into?
jbudz
(Jon Budzenski)
March 14, 2019, 5:46pm
2
Were you able to verify restarting the server brings it back down 0 (double checking UI bug vs connection leak)? Node.js should hard close any socket open more than 2 minutes so it's a tough one.
Do you have any load balancers or proxies in the setup?
davidv
(David)
March 17, 2019, 8:02am
3
On restart it sets back to 0.
No load balancers or proxies.
jbudz
(Jon Budzenski)
March 19, 2019, 3:07pm
5
@davidv I did some digging, but nothing conclusive. This is a Kibana bug, and you're not the only one that has seen it. It appears to be fixed in newer versions, but I haven't been able to find the root cause, or where it was fixed. We have two major upgrades to our web server framework in between, and that's the most suspect area.
It may be worthwhile to check if this is the actual number of open connections, or a display bug. Does lsof -u <kibana_user>
have a huge list of connections?
davidv
(David)
March 20, 2019, 10:10am
6
@jbudz could you share the existing bugs?
lsof output is:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
node 2631 kibana cwd DIR 253,0 4096 2 /
node 2631 kibana rtd DIR 253,0 4096 2 /
node 2631 kibana txt REG 253,0 30559647 24707090 /usr/share/kibana/node/bin/node
node 2631 kibana mem REG 253,0 161776 12968364 /lib64/ld-2.12.so
node 2631 kibana mem REG 253,0 23088 12968372 /lib64/libdl-2.12.so
node 2631 kibana mem REG 253,0 1930416 12968365 /lib64/libc-2.12.so
node 2631 kibana mem REG 253,0 146592 12968370 /lib64/libpthread-2.12.so
node 2631 kibana mem REG 253,0 47760 12968377 /lib64/librt-2.12.so
node 2631 kibana mem REG 253,0 600048 12968366 /lib64/libm-2.12.so
node 2631 kibana DEL REG 253,0 12968369 /lib64/libgcc_s-4.4.7-20120601.so.1;5c8a6fec
node 2631 kibana DEL REG 253,0 24512673 /usr/lib64/libstdc++.so.6.0.13
node 2631 kibana mem REG 253,0 66432 12967968 /lib64/libnss_files-2.12.so
node 2631 kibana 0r CHR 1,3 0t0 4601 /dev/null
node 2631 kibana 1w REG 253,0 1076205730 10584157 /var/log/kibana/kibana.stdout
node 2631 kibana 2w REG 253,0 168323847 10584158 /var/log/kibana/kibana.stderr
node 2631 kibana 3r FIFO 0,8 0t0 15388 pipe
node 2631 kibana 4w FIFO 0,8 0t0 15388 pipe
node 2631 kibana 5u REG 0,9 0 4597 [eventpoll]
node 2631 kibana 6r FIFO 0,8 0t0 15389 pipe
node 2631 kibana 7w FIFO 0,8 0t0 15389 pipe
node 2631 kibana 8u REG 0,9 0 4597 [eventfd]
node 2631 kibana 9u IPv4 3429020 0t0 TCP ElasticF:52154->ElasticF:wap-wsp (ESTABLISHED)
node 2631 kibana 10r CHR 1,3 0t0 4601 /dev/null
node 2631 kibana 11u IPv4 3429021 0t0 TCP ElasticF:52156->ElasticF:wap-wsp (ESTABLISHED)
node 2631 kibana 12u IPv4 16917 0t0 TCP *:esmagent (LISTEN)
node 2631 kibana 13u IPv4 3427395 0t0 TCP ElasticF:51980->ElasticF:wap-wsp (ESTABLISHED)
node 2631 kibana 14u IPv4 3425786 0t0 TCP ElasticF:51812->ElasticF:wap-wsp (ESTABLISHED)
node 2631 kibana 15u IPv4 3427396 0t0 TCP ElasticF:51982->ElasticF:wap-wsp (ESTABLISHED)
node 2631 kibana 16u IPv4 3425891 0t0 TCP ElasticF:51828->ElasticF:wap-wsp (ESTABLISHED)
node 2631 kibana 17u IPv4 3530303 0t0 TCP ElasticF:59306->ElasticF:wap-wsp (ESTABLISHED)
node 2631 kibana 18u IPv4 3793098 0t0 TCP 10.65.151.75:esmagent-141.137.198.153:64441 (ESTABLISHED)
node 2631 kibana 19u IPv4 3809909 0t0 TCP ElasticF:49064->ElasticF:wap-wsp (ESTABLISHED)
node 2631 kibana 20u IPv4 3639689 0t0 TCP ElasticF:38646->ElasticF:wap-wsp (ESTABLISHED)
node 2631 kibana 21u IPv4 3809910 0t0 TCP ElasticF:49066->ElasticF:wap-wsp (ESTABLISHED)
node 2631 kibana 22u IPv4 3426189 0t0 TCP ElasticF:51868->ElasticF:wap-wsp (ESTABLISHED)
node 2631 kibana 24u IPv4 3425981 0t0 TCP ElasticF:51840->ElasticF:wap-wsp (ESTABLISHED)
node 2631 kibana 26u IPv4 3429024 0t0 TCP ElasticF:52160->ElasticF:wap-wsp (ESTABLISHED)
node 2631 kibana 27u IPv4 3643016 0t0 TCP ElasticF:38868->ElasticF:wap-wsp (ESTABLISHED)
node 2631 kibana 28u IPv4 3426195 0t0 TCP ElasticF:51874->ElasticF:wap-wsp (ESTABLISHED)
node 2631 kibana 29u IPv4 3643017 0t0 TCP ElasticF:38870->ElasticF:wap-wsp (ESTABLISHED)
node 2631 kibana 31u IPv4 3643019 0t0 TCP ElasticF:38872->ElasticF:wap-wsp (ESTABLISHED)
node 2631 kibana 32u IPv4 3639704 0t0 TCP ElasticF:38652->ElasticF:wap-wsp (ESTABLISHED)
system
(system)
Closed
April 17, 2019, 10:10am
7
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.