Can't get rsyslog+ELK working

Elastic Stack Logstash
#1

Hello,

I am trying to setup a rsyslog+ELK stack following this tutorial without success:

My configuration is similar to the one in the tutorial except that the rsyslog server and the whole ELK stack are running on the same server.
Note that the ELK stack was previously used with Filebeat and it was working fine.

The rsyslog server receive the logs from the client and generate the template as expected. It seems to also send the logs to logstash properly according to rsyslog debug logs:
"ACTION 0x1bc5a40 [builtin:omfwd:@localhost:10514;json-template]"
Full rsyslog server logs generated when "sudo ls" is executed on the rsyslog client are available here:
https://www.zerobin.net/?1cc8d26e568ae70f#EAnAGTD+pZUMgh0U8Ff7E2jCWU3/vSdZOs4KxXP33aU=

However, I can't find the logs in elasticsearch and I don't see anything happening in logstash logs, even though it is listening to the correct port:
$ sudo netstat -na | grep 10514
udp6 0 0 127.0.0.1:10514 :::*

Here are the versions of the softs I am using:
Ubuntu 14.04.4
rsyslog 7.4.4
logstash 2.2.4
elasticsearch 2.3.3

The configuration I use for rsyslog, logstash and elasticsearch can be found here:
https://www.zerobin.net/?54aa190f432da0de#Ci72wJ8z1koLEK41S1A6MChtbDh+gPdphDll5Pc0Hc8=

Does anyone have an idea about how to find what is wrong ?

Thanks !

#2

If you shut down Logstash and use e.g. netcat to listen on localhost:10514, do you get anything from rsyslog?

#3

That was a good thing to test, I get nothing with "netcat -ulv 10514" so it looks like it might be a rsyslog issue/misconfiguration after all but I still don't see what is wrong since the configuration remain the same as the one in the tutorial.

Just in case I checked iptables to see if it wasn't configured to block some ports but everything is fine:
$ sudo iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

#4

Sniffing the what is happening on the 10514 port was indeed the way to find what was wrong ! It seems that "localhost" got translated into the IPv6 address instead of the IPv4 one which ended up with the packets being dropped:

$ sudo tcpdump -vvv -i lo -n udp dst port 10514
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
15:37:17.695057 IP6 (hlim 64, next-header UDP (17) payload length: 254) ::1.51948 > ::1.10514: [bad udp cksum 0x0111 -> 0x1804!] UDP, length 246
15:37:17.695987 IP6 (hlim 64, next-header UDP (17) payload length: 252) ::1.51948 > ::1.10514: [bad udp cksum 0x010f -> 0x91f4!] UDP, length 244
15:37:17.696827 IP6 (hlim 64, next-header UDP (17) payload length: 236) ::1.51948 > ::1.10514: [bad udp cksum 0x00ff -> 0xdb09!] UDP, length 228

Replacing "localhost" with "127.0.0.1" in /etc/rsyslog.d/60-output.conf solves the issue and now everything is working fine !

Thank you Magnus Bäck !

#5