Logstash will not start with netflow module

[jkevan (Josh)]

I have Logstash installed as a service. It starts as root so i can bind privileged ports.
i successfully ran the

bin/logstash --modules netflow --setup -M "netflow.var.kibana.host=10.x.x.x:5601" -M "netflow.var.input.udp.port=9996" -M "netflow.var.elasticsearch.hosts=10.x.x.x:9200"

to get the dashboards and index pattern setup. I then put the module configs in the logstash.yml:

modules:
  - name: netflow
    var.elasticsearch.hosts: "10.x.x.x:9200"
    var.kibana.host: "10.x.x.x:5601"
    var.input.udp.port: 9996

and started her up... the module starts to load but then hangs on

[2018-06-15T15:15:32,382][INFO ][logstash.filters.translate] refreshing dictionary file
[2018-06-15T15:15:32,410][INFO ][logstash.filters.translate] refreshing dictionary file
[2018-06-15T15:15:32,411][INFO ][logstash.filters.translate] refreshing dictionary file
[2018-06-15T15:15:32,443][INFO ][logstash.filters.translate] refreshing dictionary file
[2018-06-15T15:20:32,633][INFO ][logstash.filters.translate] refreshing dictionary file
[2018-06-15T15:20:32,663][INFO ][logstash.filters.translate] refreshing dictionary file
[2018-06-15T15:20:32,693][INFO ][logstash.filters.translate] refreshing dictionary file
[2018-06-15T15:20:32,721][INFO ][logstash.filters.translate] refreshing dictionary file
[2018-06-15T15:20:32,748][INFO ][logstash.filters.translate] refreshing dictionary file
[2018-06-15T15:20:32,773][INFO ][logstash.filters.translate] refreshing dictionary file

is this a permission issue because its being run as root? or is it something deeper?

Thanks

[jkevan (Josh)]

nevermind it looks like it started successfully, but it only started the one (netflow) pipeline... all my other pipelines were not started.

[rcowart (Rob Cowart)]

The Logstash modules will take over your Logstash configuration, and cannot be setup in combination with other Pipelines. This is a known issue.

A few points that might be relevant for you:

1. The Logstash Netflow Module was originally based on ElastiFlow 1.0.0 and is quite dated at this point. You might want to consider using the latest release of ElastiFlow instead. You can see a comparison of the differences HERE. You will notice from the setup instructions that ElastiFlow is designed to be setup in pipelines.yml and fully supports running along side other pipelines. However...

2. Just because you can run ElastiFlow in a multi-pipeline setup. Network flow data can be very voluminous. This is easily underestimated. You will want to be certain that you have sufficient resources available to handle the amount of data you will collect. You might find it necessary to give Flow collection it own dedicated Logstash instance, tuned to the needs of high UDP ingest rates.

[jkevan (Josh)]

Great information! Thank you! I have decided to keep my elastiflow monitoring on it's own instance, as after a day of collecting logs from just one core I was pulling in 13GB of logs in one day.