Logstash Module Netflow

hispeed · April 7, 2019, 11:14am

Hi,

I want to load the netflow module on Centos 7 with Java 8 and ELK V.7.0 RC 2.
All services are up and everything looks fine. I had a little bit trouble with Java because I wanted to use Java 11.

Now when I want to load the netflow module:

sudo /usr/share/logstash/bin/logstash --modules netflow --setup --path.settings /etc/logstash

> [root@SVGWMA-ELK-TEST-01 usr]# sudo /usr/share/logstash/bin/logstash --modules netflow --setup --path.settings /etc/logstash
> Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
> [2019-04-07T13:07:36,278][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
> [2019-04-07T13:07:36,308][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.0.0"}
> [2019-04-07T13:07:37,821][INFO ][logstash.config.modulescommon] Setting up the netflow module
> [2019-04-07T13:07:38,634][ERROR][logstash.modules.kibanaclient] Error when executing Kibana client request {:error=>#<Manticore::SocketException: Verbindungsaufbau abgelehnt (Connection refused)>}
> [2019-04-07T13:07:38,926][ERROR][logstash.modules.kibanaclient] Error when executing Kibana client request {:error=>#<Manticore::SocketException: Verbindungsaufbau abgelehnt (Connection refused)>}
> [2019-04-07T13:07:39,161][ERROR][logstash.config.sourceloader] Could not fetch all the sources {:exception=>LogStash::ConfigLoadingError, :message=>"Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: [\"localhost:9200\"] and Kibana hosts: [\"localhost:5601\"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:108:in `block in pipeline_configs'", "org/jruby/RubyArray.java:1792:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:54:in `pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/modules.rb:14:in `pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:61:in `block in fetch'", "org/jruby/RubyArray.java:2572:in `collect'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:60:in `fetch'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:148:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:362:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
> [2019-04-07T13:07:39,169][ERROR][logstash.agent           ] An exception happened when converging configuration {:exception=>RuntimeError, :message=>"Could not fetch the configuration, message: Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: [\"localhost:9200\"] and Kibana hosts: [\"localhost:5601\"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/agent.rb:155:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:362:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
> [2019-04-07T13:07:39,741][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

hispeed · April 7, 2019, 3:56pm

Can be closed.

I fixed it with restart and give the correct path with it. As well as configure logstash.yml correctly.

Habitz · April 22, 2019, 3:34pm

hi hispeed,

I'm having a similar issue, could you please share the configuration you applied? Thanks!

hispeed · April 22, 2019, 5:19pm

Hi Rodrigo,

The problem is probably in elasticsearch.yml. Your Elasticsearch server is probably running in development mode and not in "productive mode". I had to correct network.host: 0.0.0.0 this because it's running on the same machine and it's no cluster.
Nobody was telling me that but this was the reason filebeat and netflow couldn't create the sample dashboard. Because this is not working properly you get into other problems. In my opinion elastic.co should write about that a proper tutorial how that works in V.7.0. This problem is caused because 99% of all tutorials online are not working properly. I have mentioned this problem several times.

> # ----------------------------------- Memory -----------------------------------
> #
> # Lock the memory on startup:
> #
> bootstrap.memory_lock: true
> #
> # Make sure that the heap size is set to about half the memory available
> # on the system and that the owner of the process is allowed to use this
> # limit.
> #
> # Elasticsearch performs poorly when the system is swapping the memory.
> #
> # ---------------------------------- Network -----------------------------------
> #
> # Set the bind address to a specific IP (IPv4 or IPv6):
> #
> network.host: 0.0.0.0
> #
> # Set a custom port for HTTP:
> #
> http.port: 9200
> #
> # For more information, consult the network module documentation.

Habitz · April 24, 2019, 2:23pm

Hi hispeed,

Thanks for the answer, nevertheless I still have the same issue. This is what my config looks like:

modules:

name: netflow
var.input.udp.port: 2055
var.elasticsearch.hosts: http://127.0.0.1:9200
var.elasticsearch.ssl.enabled: false
var.kibana.host: 127.0.0.1:5601
var.kibana.scheme: http
var.kibana.ssl.enabled: false
var.kibana.ssl.verification_mode: disable

Am I missing something? Thanks!

hispeed · April 24, 2019, 5:20pm

Hi Habitz,

I have:

modules:
  - name: netflow
    var.input.udp.port: 2055

For loading the templates i used:

sudo /usr/share/logstash/bin/logstash --modules netflow --setup -M netflow.var.input.udp.port=2055 -M netflow.var.elasticsearch.hosts="127.0.0.1:9200" -M netflow.var.kibana.host="127.0.0.1:9200"

hispeed · April 24, 2019, 7:41pm

Habitz please look at: Can't recieve filebeat logs via Logstash
This will also infect you and I hope i get a step by step guide because there are several problems if you don't setup it correctly.

Habitz · April 27, 2019, 5:24pm

thanks mate ... I will check that out!

Jesse_Bacon · April 29, 2019, 5:05pm

Updates?
I would like to use Netflow for tier 1 and FPC for tier 2 staff. IS there anything that will allow the collection of Netflow data using the Logstash system? I wanted to use the dashboards, they look nice. I checked the modules directory and found the raw JSON files as well. Are there instructions for installing these manually? I get the following error (IP's and ports are correct):

ERROR] 2019-04-29 12:17:16.559 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] sourceloader - Could not fetch all the sources {:exception=>LogStash::ConfigLoadingError, :message=>"Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: ["192.168.56.101:9200"] and Kibana hosts: ["192.168.56.101:5601"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:108:in block in pipeline_configs'", "org/jruby/RubyArray.java:1792:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:54:in pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/modules.rb:14:inpipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:61:in block in fetch'", "org/jruby/RubyArray.java:2572:incollect'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:60:in fetch'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:148:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:362:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

[ERROR] 2019-04-29 12:17:16.564 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - An exception happened when converging configuration {:exception=>RuntimeError, :message=>"Could not fetch the configuration, message: Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: ["192.168.56.101:9200"] and Kibana hosts: ["192.168.56.101:5601"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/agent.rb:155:in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:362:in block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/task.rb:24:inblock in initialize'"]}

[INFO ] 2019-04-29 12:17:16.711 [LogStash::Runner] runner - Logstash shut down.

hispeed · April 30, 2019, 7:51pm

Hi Jesse_Bacon

No updates anymore in this thread. I assume that the problems ar comming from am faulty configurated ELK Stack or from a wrong configurated logstash or wrong configured Netflow Module plugin. Please have a look at: Can't recieve filebeat logs via Logstash

Jesse_Bacon · April 30, 2019, 8:30pm

Thanks hispeed,

I assume they want us to switch to Packet Beat. I corrected the elastic ip defaults to local host issue and it still will not push the dashboards and create the index pattern. There is something blocking it. I went ahead and built a SiLK box. I’ll just grab the data in python and convert it to csv. I will have to build my own dashboards though.

V/R,

Jesse Bacon

system · May 28, 2019, 8:30pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.