Help troubleshooting tcp input

[reswob]

I have a simple conf file:

input {
     tcp {
         port => 2000
     }
 }

filter {
 }

output {
          stdout { coded => rubydebug }
 }

That's it. I'm using nifi putTCP to send the logs. I see the logs come in via tcpdump, but nothing shows up on the screen.

If I switch the above config to use input - udp and nifi to use putUDP, all the logs show up.
But nothing with input - tcp and nifi putTCP

There are not errors in the logstash log either.

iptables is stopped and firewalld isn't installed.

Thoughts? suggestions?

[Christian_Dahlqvist]

Could it be that entries coming via TCP are not delimited by newlines?

[Badger]

Verify that if you have an IP route configured to the source address of the packets, that it points to the interface on which the packets arrive. The network stack can ignore packets that arrive on another interface.

[reswob]

OK, here's what I've learned.

First @Christian_Dahlqvist, the entries are delimited by newlines, I did check.
And @Badger, I'm sure there is an IP route back.

So to test, I enabled tcp receive on rsyslog, then set rsyslog to write all received logs to a file. This worked, all logs sent were written to the file. Then I went back to LS and tried again. I added

file {
    path => "/opt/testfolder/test.txt"
}

to the conf file and fired LS again. In a separate window I ran 'netstat -lnatpec' and watched to see if a: LS was listening on the assigned port (it was) and b: if a connection was established (again, it did).
But nothing written to the screen and nothing written to the file.

LS was running in debug mode so I checked the log and saw lots of this:

[DEBUG][logstash.codecs.line         ] config LogStash::Codecs::Line/@id = "line_<UUID>"
[DEBUG][logstash.codecs.line         ] config LogStash::Codecs::Line/@enable_metric = true
[DEBUG][logstash.codecs.line         ] config LogStash::Codecs::Line/@charset = "UTF-8"
[DEBUG][logstash.codecs.line         ] config LogStash::Codecs::Line/@delimiter = "\n"

which repeated multiple times until there was an entry:

[DEBUG][logstash.pipeline              ] Pushing flush onto pipeline

where upon the cycle would repeat itself. So it SEEMS like LS is working normally, but yet I get nothing printed to screen and nothing to the file.

Am I interpreting the above right? Or do those logs point to something wrong?

BTW, I'm running LS v5.6.5

[reswob]

So I figured it out and, yup, a little bit of pebkac. Turns out the NGFW between the two devices was only partially open. The two rules that created the confusion were

1. allow DHCP traffic out on any any (wha??? I didn't put that there)
2. syslog traffic to the specific LS server

So that's why I always saw traffic getting to the LS server, it used the DHCP rule. But the NGFW allowed some things through but not others that it didn't think were DHCP based on the protocols it saw. On top of that, it didn't log any traffic as being denied so there were no red flags indicating what could be wrong.

So when I fixed the rules to exactly allow the traffic I wanted, things worked.