Logstash Netflow Docs Out of Date?


(Jarrad) #1

Hi.

I've been following the docs here: https://www.elastic.co/guide/en/logstash/6.x/netflow-module.html for an ELK stack running ELK 6.2.3 I've built today running on Ubuntu 16.04 server. I'd like to get Netflow integration but the documentation is either out of date or broken - it doesn't work as per the docs.

My setup is really insecure and basic. No X-pack, no ssl, no usernames, no passwords. This was my last attempt. 10.254.254.220 is the IP address of the ELK node.

user@elk:/usr/share/logstash# bin/logstash --modules netflow --setup -M netflow.var.input.udp.port=2055 -M netflow.var.elasticsearch.hosts="10.254.254.220:9200" -M netflow.var.kibana.host="10.254.254.220:5601"

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2018-04-08 19:10:32.492 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[INFO ] 2018-04-08 19:10:32.499 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[WARN ] 2018-04-08 19:10:32.867 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2018-04-08 19:10:33.017 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.3"}
[INFO ] 2018-04-08 19:10:33.151 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9601}
[ERROR] 2018-04-08 19:10:33.802 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] sourceloader - Could not fetch all the sources {:exception=>LogStash::ConfigLoadingError, :message=>"Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: ["10.254.254.220:9200"] and Kibana hosts: ["10.254.254.220:5601"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:99:in block in pipeline_configs'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:56:in pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/modules.rb:16:inpipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:59:in block in fetch'", "org/jruby/RubyArray.java:2481:incollect'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:58:in fetch'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:148:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in block in initialize'"]} [ERROR] 2018-04-08 19:10:33.803 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - An exception happened when converging configuration {:exception=>RuntimeError, :message=>"Could not fetch the configuration, message: Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: [\"10.254.254.220:9200\"] and Kibana hosts: [\"10.254.254.220:5601\"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/agent.rb:155:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

If I am reading this correctly, Logstash is fine but it can't connect to Elastic or Kibana - which is weird because they are both 100% running.

I'm going to go over some of these "fixes" again but would love some help. Is it an issue with ruby perhaps?

Thanks


(Robert Cowart) #2

I don't know whether the docs are out of date, but arguably the Netflow module itself. The Logstash Netflow module was based on v1.0.0 of ElastiFlow, which is now on v2.1.0.


(Jarrad) #3

Well, I managed to discover at least one thing mucking around again.

This works, kind of.

user@elk:/usr/share/logstash# bin/logstash --modules netflow --setup -M "netflow.var.input.udp.port=2055" -M "netflow.var.kibana.host=10.254.254.220:5601"

It does what it is supposed to do (netflows are not being stored and made available) but rather than running setup and finishing off its hanging a process open to keep it all running.

I didn't think that is what a setup would be expected to do?

As to Robert - thank you. That certainly makes a lot more sense...


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.