Netflow module and *.conf file

[ggdagg]

Hi,

I set up the Netflow module for Logstash and it works well. In my Logstash config (in /etc/logstash/logstash.yml), I added:
modules:
- name: netflow
var.input.udp.port: 2055
var.kibana.host: "mykibanahost.com:5601"
var.elasticsearch.hosts: "myelasticsearchhost.com:9200"

In parallel, I set up Syslog UDP input following this documentation but it looks like when Netflow module is set up and enable, Logstash doesn't take my *.conf file (located in my case in /etc/logstash/conf.d) in consideration and doesn't open the port 5000. Only Starting UDP listener {:address=>"0.0.0.0:2055"} is seen in the log.
If I comment out my module section in /etc/logstash/logstash.yml, the Syslog port 5000 starts to listen correctly.

Hence my question: is it possible to open multiple UDP ports via Logstash? If so, how can I open both the Netflow port and the Syslog port on the same system?

Logstash: 6.2.2
Elasticsearch: 6.2.2
Kibana: 6.2.2
System: CentOS 7

Thank you for your help.

[rcowart (Rob Cowart)]

First I wanted to share some information about Netflow. If you are serious about collecting Netflow (as well as sFlow and IPFIX) with the Elastic Stack, you may want to consider ElastiFlow: https://github.com/robcowart/elastiflow

The Logstash Netflow Module was actually based on v1.0.0 of ElastiFlow (it is basically just v1.0.0 implemented as a Logstash Module). However ElastiFlow is now at v2.2.0, and the Master branch has some updates that will become 2.3 over the weekend (so if you try it, use the master branch). You can review the notes for each release to see what has changed since v1.0.0.

If you are just getting started and want a head start on basic syslog handling. Have a look at this...

koiossian/synesis_lite_syslog

The solutions are designed to work well with the multi-pipeline capabilities introduced in 6.x. Assuming your logstash configuration is in /etc/logstash you would put the elastiflow and synesis_lite_syslog directories in /etc/logstash.

Then edit /etc/logstash/pipelines.yml to look like this...

- pipeline.id: elastiflow
  path.config: "/etc/logstash/elastiflow/conf.d"

- pipeline.id: synesis_lite_syslog
  path.config: "/etc/logstash/synesis_lite_syslog/conf.d"

All configuration of these solutions is done via environment variables. Once these are setup, simply start Logstash and it will load both pipelines and you will be able to collect and visualize Netflow, sFlow, IPFIX and Syslog data.

Rob

Robert Cowart (rob@koiossian.com)
www.koiossian.com
True Turnkey SOLUTIONS for the Elastic Stack

[ggdagg]

Hi Rob,

Thank you for your reply and the information related to the Netflow module in Logstash. Indeed, I didn't know that this module was based on ElastiFlow, so I might switch to it for my tests. However, I'm still don't know if I will use my ELK for Netflow, it's only for test purposes for now. Thanks also for the link to the Synesis lite Syslog.
I will try to test both on my test environment and see if it can remove the limitation I had initially.