Hi.
I've been following the docs here: https://www.elastic.co/guide/en/logstash/6.x/netflow-module.html for an ELK stack running ELK 6.2.3 I've built today running on Ubuntu 16.04 server. I'd like to get Netflow integration but the documentation is either out of date or broken - it doesn't work as per the docs.
- I've tried the solves from here:
- Netflow Setup Broken? - path.config is already commented out in my build
- This one: Netflow Setup Broken? - adding the module stuff to logstash.yml and just trying to run --modules netflow --setup doesn't work.
- Nor does it here either: Run netflow moudules failed
- This one: Kibanaclient - Error when executing Kibana client request - Just doesn't work outright
- And this made no impact either: Run "---setup" netflow module when LS is a service (was Install netflow module error)
My setup is really insecure and basic. No X-pack, no ssl, no usernames, no passwords. This was my last attempt. 10.254.254.220 is the IP address of the ELK node.
user@elk:/usr/share/logstash# bin/logstash --modules netflow --setup -M netflow.var.input.udp.port=2055 -M netflow.var.elasticsearch.hosts="10.254.254.220:9200" -M netflow.var.kibana.host="10.254.254.220:5601"
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2018-04-08 19:10:32.492 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[INFO ] 2018-04-08 19:10:32.499 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[WARN ] 2018-04-08 19:10:32.867 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2018-04-08 19:10:33.017 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.3"}
[INFO ] 2018-04-08 19:10:33.151 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9601}
[ERROR] 2018-04-08 19:10:33.802 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] sourceloader - Could not fetch all the sources {:exception=>LogStash::ConfigLoadingError, :message=>"Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: ["10.254.254.220:9200"] and Kibana hosts: ["10.254.254.220:5601"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:99:in block in pipeline_configs'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:56:in pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/modules.rb:16:inpipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:59:in block in fetch'", "org/jruby/RubyArray.java:2481:incollect'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:58:in fetch'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:148:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in block in initialize'"]} [ERROR] 2018-04-08 19:10:33.803 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - An exception happened when converging configuration {:exception=>RuntimeError, :message=>"Could not fetch the configuration, message: Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: [\"10.254.254.220:9200\"] and Kibana hosts: [\"10.254.254.220:5601\"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/agent.rb:155:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
If I am reading this correctly, Logstash is fine but it can't connect to Elastic or Kibana - which is weird because they are both 100% running.
I'm going to go over some of these "fixes" again but would love some help. Is it an issue with ruby perhaps?
Thanks