So, I was today debugging this issue with tcpdump and saw that no FIN or RST was received on the logstash side. On the sending side from F5 BigIp they did not see any connection leakage and everything seems normal. So an intermediary was causing problems.
Further investigations with our network team uncovered that some packets were being blocked in the firewall. Apparently the connections was first set up as syslog, and then further down the line when switching to raw tcp the firewall was not updated. This lead to the data was being received but the FIN packets was not received as they were blocked.
After allowing any protocol on the designated port, not only syslog , the connection leak is fixed!!!