Reverse connections from lumberjack

Elastic Stack Logstash
1

Here is my setup:

  1. system-A -- running logstash-forwarder, shipping to B
  2. system-B -- running lumberjack on port 5043 and logstash-forwarder
  3. system-C -- running logstash-forwarder, shipping to B

I'm seeing reverse connections being made from system B to random ports on systems A, B and C.
If I run lsof on system B then I get follwoing:

sudo lsof -i :5043
COMMAND     PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
java       6470 root   16u  IPv4 95926184      0t0  TCP *:swxadmin (LISTEN)
java       6470 root   58u  IPv4 95926674      0t0  TCP B:swxadmin->A:54825 (ESTABLISHED)
java       6470 root   62u  IPv4 95936170      0t0  TCP B:swxadmin->C:36901 (ESTABLISHED)
java       6470 root   66u  IPv4 95936172      0t0  TCP B:swxadmin->B:58816 (ESTABLISHED)

process 6470 is the Logstash process (server component) running on system B.
Logstash version is 1.5.4 and logstash-forwarders are version 0.3.1

Does anyone know what these reverse connection are and why they're being made? I tried troubleshooting but since Logstash refuses to run without ssl, I can't dump the traffic.

Thanks.

2

The -> arrow in the lsof output doesn't indicate who opened the connection. Quoting the lsof(1) man page:

or the local and remote Internet addresses of a network file; the local host name or IP number is followed by a colon (':'), the port, ->, and the two-part remote address;

So, your lsof output indicates that your machine listens on the "swxadmin" port and has three established connections, one each to hosts A, B, and C.

3

My mistake, sorry about that. I was checking system logs and found that there are some packets dropped by iptables on system-B, for example:

Oct 12 07:08:39 system-B kernel: [4160300.596670] D[OUT-OTHER] IN= OUT=eth0 SRC=192.168.X.B DST=192.168.X.A LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=62381 DF PROTO=TCP SPT=5043 DPT=50174 WINDOW=501 RES=0x00 ACK PSH FIN URGP=0 

Oct 12 07:08:39 system-B kernel: [4160300.596670] D[OUT-OTHER] IN= OUT=eth0 SRC=192.168.X.B DST=192.168.X.C LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=62806 DF PROTO=TCP SPT=5043  DPT=45238 WINDOW=141 RES=0x00 ACK PSH FIN URGP=0

I have rules in my iptables on system-B allowing incoming connections to port 5043 and everything works fine, but as you can see above there are outgoing connections from port 5043 which are dropped by the firewall.

4

That packet isn't a new outgoing connection, it's an acknowledgement of the closure of an existing session.

5

@magnusbaeck thanks for you help.

Just want to confirm if I'm understanding this correctly

  1. Logstash-forwarder has no data to send and it issues a FIN to terminate the connection.
  2. The logstash-server issues an ACK in response to the above and sends a FIN as well which gets dropped by the firewall?
6

Yes, that's what it looks like to me.