Logstash and ElasticSearch setup not working for Netflow logs

Elastic Stack Logstash
1

Hi Folks,
I have done a basic setup/configuration for storing Netflow logs into ElasticSearch using Logstash. My logstash config looks like:

input {
    udp {
      port => 9995
      codec => netflow {
        definitions => "<my-path>/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-codec-netflow-1.0.0/lib/logstash/codecs/netflow/netflow.yaml"
        versions => [9]
      }
    }
  }

  output {
    stdout { codec => rubydebug }
      elasticsearch {
        index => "logstash-netflow9-%{+YYYY.MM.dd}"
        host => "localhost"
      }
  }

Note - right now I am trying to get all netflow logs but would be adding a condition like below in output block around elasticsearch block:

if ( [host] == "XXX.XXX.XXX.XXX" )

With above config, when I run logstash, I don't see any output from logstash on stdout or any new index in elasticsearch. But I see below messages in logstash's logs:

{:timestamp=>"2015-08-19T07:05:07.841000+0000", :message=>"No matching template for flow id 260", :level=>:warn}
{:timestamp=>"2015-08-19T07:05:08.008000+0000", :message=>"No matching template for flow id 265", :level=>:warn}

I have verified using wireshark that netflow data is indeed available on the 9995 port and also checked with network team about the same. The template records are being sent at 5 minute interval and once the template record is received the subsequent records are decoded according to the template as seen in packets captured by wireshark.

I have tried above with elasticsearch 1.3.4 and logstash 1.4.2 (basically following the steps here - http://blogs.cisco.com/security/step-by-step-setup-of-elk-for-netflow-analytics) and things didn't work i.e. no output on stdout or no new index in elasticsearch.
I then tried with latest versions as well - elasticsearch 1.7.1 and logstash 1.5.3 but don't see any data flowing from logstash to elasticsearch or stdout.

I am unable to figure out what is missing and where, given that all the things seem to be in place. Appreciate help in further investigating/resolving this issue.

Thanks.

2

I tried to define the flow set ids for which I get warning as suggested in the thread here - https://github.com/elastic/logstash/issues/2002 but have the same issue.

Does the netflow codec fail and doesn't produce data through logstash output config if it finds some flow set fields missing? I guess the remaining fields/data should get generated/indexed if some are missing.

Is this a known problem or should I create a bug for logstash?

Thanks.

3

Can someone for whom the elasticsearch + logstash + netflow codec setup is working share their setup details like versions used, configuration, system details, firewall/iptables/ports related details, netflow specific details, etc. ?

4

I thought I'd reply, as I have today also discovered this problem.

It appears as if the netflow data being sent, in my case by cisco ASA's, sends packets with flow IDs which do not have mappings in the netflow.yaml file.

There is a github discussion about this which can be found here https://github.com/logstash-plugins/logstash-codec-netflow/issues/12 The commenters on the thread discuss the setting of value, to give logstash a match for the field id being reported.

In my case I was getting flow ID warnings for flow IDs 256, 261 and 263 - they were the majority ones that I was seeing. So I tried adding the following to my netflow.yaml file, below, using the following link https://www.elastic.co/guide/en/logstash/current/plugins-codecs-netflow.html as a guide;

256:
- :string
- :netflow_v9
263:
- :string
- :netflow_v9

It was merely a shot in the dark... irrespective it didnt' work. I dont know if I have the formatting wrong, or maybe there needs to be more detail for each template or flow ID.. I've yet to find a technical reference which explains what should be here.

The nearest I've got to, is finding RFC3954 from the cisco site, which outlines the packet templat format, this was the most technical document I could find, to try and figure out the format of the data being sent to logstash.

If anyone has any suggestions - or clairifcation on whether this is a bug or not, it would be much appreciated.

5

I just managed to get Netflow data into my ELK stack on Windows. I hope this info is helpful for you in some ways.
Instead of having a Cisco router throwing Netflow into ELK, I use port mirroring to mirror traffic of the port on the switch that connects to the Cisco router. The diagram is as below:

Switch port => Server running nprobe (nprobe Collector mode) => Logstash => ES.
Kibana is used to search and draw graphs.

  • nprobe (from nTop)
    nprobe is running using this config

    nprobe.exe /c -b 1 -V 9 -i 1 -n none --json-labels --tcp 192.168.1.x:5544 -T "%IPV4_SRC_ADDR %L4_SRC_PORT %IPV4_DST_ADDR %L4_DST_PORT %L7_PROTO_NAME %IN_BYTES %OUT_BYTES %IN_PKTS %OUT_PKTS %IN_SRC_MAC %OUT_DST_MAC %PROTOCOL"

I decide to export only useful fields to Logstash. 192.168.1.x:5544 is the IP of the ELK stack and 5544 is the tcp port Logstash is listening to.

  • Logstash
    Below is logstash config file:

      input {
tcp {
  port => 5544
}

    }

    filter {
    json {
    source => "message"
    }
    }

    output {
    elasticsearch {
    host => "localhost"
    index => "netflow-%{+YYYY.MM.dd}"
    protocol => "http"
    }
    }

Netflow codec is not used. With this config file, you can start ELK stack. In Kibana, you should see all the fields exported from nprobe.

6

Thanks for sharing your setup details.
I worked around the problem by not using Logstash in my implementation. I used nfcapd and nfdump for capturing the logs and indexing the fields of interest into ElasticSearch. So my setup looks like:

"nfcapd background service" => cron job("parse using nfdump" => "index into elastic search") => kibana

Since couple of folks apart from me have reported this issue, I have filed a bug in github for this issue - https://github.com/elastic/logstash/issues/4165

7

anyone found a way to get logstash working with this?

8

The reply from @secureict does not help?