[SOLVED] Stuck on getting netflow data into logstash

Elastic Stack Logstash
1

Hello,

I've been playing with this for longer than I would care to admit(weeks). I am trying to setup up an nProbe/Netflow into ELK and have it in one or more docker containers. I have tried the existing projects, but most are out of date now and have fields that don't exist anymore in the configuration files. It doesn't necessarily need to be docker if It would work on host, but I'm not having luck either way.

If I set up the individual official docker containers for Logstash, ElasticSearch and Kibana, I can manually run logstash inside the container and get it to pass data through to Kibana. However, for the life of me, running it on the host or in a docker container, I can't figure out how to get logstash to take in the netflow data.

I'm sure it must be something stupid I'm missing, but I would appreciate any help.

Info:
4 core, 4 GB vps running ubuntu 14.04
running docker 1.11.2 on a 4.2.0-36 host.
I've tried wiping out and clearing all iptables rules and policies with everything set to accept.
docker images are all offical repos, running latest as of today and using the default commands given.
for logstash: https://hub.docker.com/_/logstash/
and I've been running: docker run -it --rm -v "$PWD":/config-dir logstash logstash -f /config-dir/logstash.conf from the config-dir directory with my attempts at logstash.conf in it.
for input I've been setting it to udp on port 5656.
for output hosts => 127.0.0.1:9200 has been working.

I'd include exact files, but they are both small, wrong and pulled from old tutorials. So I expect they'd just get in the way at this point.

Oh, and I can confirm through tcpdump that the netflow packets are arriving via udp port 5656 & 9200(i've tried both ports at different points). It responds back with ICMP "The hostname" udp port 9200 unreachable, length 204 even with iptables cleared an on default accept.

Here is one of my probe container: docker run -d --net=host --name nprobe9200 -t -i nprobe-docker nprobe -n "the targethostname":9200 -i eth0 --verbose 2 -s 1024

If anyone could provide a working logstash config, and maybe a docker build line that is known working, I would be forever in your debt.

Thank you.

2

I think you're trying to do too much at once so when things don't work you don't know where to start. Start by getting Logstash running in one container and have it emit received logs via a stdout { codec => rubydebug } output. Depending on how the container is set up, those logs may or may not end up in the container's logs. Does that work? If not, how are you firing up the container and what does your Logstash configuration look like?

3

It ended up being iptables, while I had a rule in the DOCKER chain, I needed to put one in the nat table before it would work.

Ended up using iptables -t nat -I DOCKER -p udp -m udp --dport 5656 -j DNAT --to-destination 172.17.0.3:5656 ← ip of logstash container.

4

It ended up being iptables, while I had a rule in the DOCKER chain, I
needed to put one in the nat table before it would work.

Ended up using iptables -t nat -I DOCKER -p udp -m udp --dport 5656 -j DNAT
--to-destination 172.17.0.3:5656 ← ip of logstash container.