Succesfully configured logstash 5.6 with netflow setup for remote Kibana and remote Elasticsearch server as follows:
bin/logstash --path.settings /etc/logstash --modules netflow --setup -M netflow.var.input.udp.port=7000 -M var.elasticsearch.hosts="xxx.xxx.xxx.134:5601" -M var.kibana.host="xxx.xxx.xx.133:9200"
and in my /etc/logstash.yml file:
var.PLUGIN_TYPE.PLUGIN_NAME.KEY
modules:
- name: netflow
var.input.udp.port: 7000
var.elasticsearch.hosts: "xxx.xxx.xxx.134:9200"
var.kibana.host: "xxx.xxx.xxx.133:5601"
Tried to install a second instance on a new machine, but with freshly downloaded logstash 6.0:
bin/logstash --path.settings /etc/logstash --modules netflow --setup -M netflow.var.input.udp.port=7000 -M var.elasticsearch.hosts="xxx.xxx.xxx.134:5601" -M var.kibana.host="xxx.xxx.xx.133:9200"
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
ERROR: Settings 'path.config' (-f) or 'config.string' (-e) can't be used in conjunction with (--modules) or the "modules:" block in the logstash.yml file.
usage:
bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
bin/logstash --help
The error appears to be a bug - if I manually try setting the netflow vars on the command line:
bin/logstash --modules netflow --setup -M "var.kibana.host=xxx.xxx.xxx.133:5601" -M "var.elasticsearch.hosts=xxx.xxx.xxx.134:9200"
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[ERROR] 2017-11-16 16:20:38.744 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] kibanaclient - Error when executing Kibana client request {:error=>#<Manticore::SocketException: Connection refused (Connection refused)>}
[ERROR] 2017-11-16 16:20:38.758 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] kibanaclient - Error when executing Kibana client request {:error=>#<Manticore::SocketException: Connection refused (Connection refused)>}
[ERROR] 2017-11-16 16:20:38.785 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] sourceloader - Could not fetch all the sources {:exception=>LogStash::ConfigLoadingError, :message=>"Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: ["localhost:9200"] and Kibana hosts: ["localhost:5601"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:99:in block in pipeline_configs'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:56:in pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/modules.rb:16:inpipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:59:in block in fetch'", "org/jruby/RubyArray.java:2481:incollect'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:58:in fetch'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:148:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:362:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in block in initialize'"]} [ERROR] 2017-11-16 16:20:38.786 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - An exception happened when converging configuration {:exception=>RuntimeError, :message=>"Could not fetch the configuration, message: Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: [\"localhost:9200\"] and Kibana hosts: [\"localhost:5601\"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/agent.rb:155:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:362:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
It assumes localhost and -obviously- fails.
Help?
