Here is my setup:
- system-A -- running logstash-forwarder, shipping to B
- system-B -- running lumberjack on port 5043 and logstash-forwarder
- system-C -- running logstash-forwarder, shipping to B
I'm seeing reverse connections being made from system B to random ports on systems A, B and C.
If I run lsof on system B then I get follwoing:
sudo lsof -i :5043
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 6470 root 16u IPv4 95926184 0t0 TCP *:swxadmin (LISTEN)
java 6470 root 58u IPv4 95926674 0t0 TCP B:swxadmin->A:54825 (ESTABLISHED)
java 6470 root 62u IPv4 95936170 0t0 TCP B:swxadmin->C:36901 (ESTABLISHED)
java 6470 root 66u IPv4 95936172 0t0 TCP B:swxadmin->B:58816 (ESTABLISHED)
process 6470 is the Logstash process (server component) running on system B.
Logstash version is 1.5.4 and logstash-forwarders are version 0.3.1
Does anyone know what these reverse connection are and why they're being made? I tried troubleshooting but since Logstash refuses to run without ssl, I can't dump the traffic.
-> arrow in the lsof output doesn't indicate who opened the connection. Quoting the lsof(1) man page:
or the local and remote Internet addresses of a network file; the local host name or IP number is followed by a colon (':'), the port,
->, and the two-part remote address;
So, your lsof output indicates that your machine listens on the "swxadmin" port and has three established connections, one each to hosts A, B, and C.
My mistake, sorry about that. I was checking system logs and found that there are some packets dropped by iptables on system-B, for example:
Oct 12 07:08:39 system-B kernel: [4160300.596670] D[OUT-OTHER] IN= OUT=eth0 SRC=192.168.X.B DST=192.168.X.A LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=62381 DF PROTO=TCP SPT=5043 DPT=50174 WINDOW=501 RES=0x00 ACK PSH FIN URGP=0
Oct 12 07:08:39 system-B kernel: [4160300.596670] D[OUT-OTHER] IN= OUT=eth0 SRC=192.168.X.B DST=192.168.X.C LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=62806 DF PROTO=TCP SPT=5043 DPT=45238 WINDOW=141 RES=0x00 ACK PSH FIN URGP=0
I have rules in my iptables on system-B allowing incoming connections to port 5043 and everything works fine, but as you can see above there are outgoing connections from port 5043 which are dropped by the firewall.
That packet isn't a new outgoing connection, it's an acknowledgement of the closure of an existing session.
@magnusbaeck thanks for you help.
Just want to confirm if I'm understanding this correctly
- Logstash-forwarder has no data to send and it issues a FIN to terminate the connection.
- The logstash-server issues an ACK in response to the above and sends a FIN as well which gets dropped by the firewall?
Yes, that's what it looks like to me.