Was hoping someone might be able to advise.
Running: latest versions of esearch, kibana, and logstash installed via repo plus Oracle Java 1.8.0_181 on a Ubuntu Server 18.04 VM.
I'm trying to set up an ELK stack for netflow, but when running the following command a lot of errors are produced and the process is clearly failing.
as root: /usr/share/logstash/bin/logstash --modules netflow --setup -M netflow.var.input.udp.port=2055
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2018-09-23 14:44:50.001 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2018-09-23 14:44:50.024 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[WARN ] 2018-09-23 14:44:50.681 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2018-09-23 14:44:50.764 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"b76cbfb0-1367-46fd-ab15-152e23340407", :path=>"/usr/share/logstash/data/uuid"}
[INFO ] 2018-09-23 14:44:51.878 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.4.1"}
[INFO ] 2018-09-23 14:44:52.161 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] modulescommon - Setting up the netflow module
[ERROR] 2018-09-23 14:44:53.468 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] kibanaclient - Error when executing Kibana client request {:error=>#<Manticore::UnknownException: Unrecognized SSL message, plaintext connection?>}
[ERROR] 2018-09-23 14:44:53.747 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] kibanaclient - Error when executing Kibana client request {:error=>#<Manticore::UnknownException: Unrecognized SSL message, plaintext connection?>}
[ERROR] 2018-09-23 14:44:54.114 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] sourceloader - Could not fetch all the sources {:exception=>LogStash::ConfigLoadingError, :message=>"Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: [\"localhost:9200\"] and Kibana hosts: [\"localhost:5601\"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:108:in `block in pipeline_configs'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:54:in `pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/modules.rb:14:in `pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:61:in `block in fetch'", "org/jruby/RubyArray.java:2481:in `collect'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:60:in `fetch'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:142:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:93:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:362:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
[ERROR] 2018-09-23 14:44:54.137 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - An exception happened when converging configuration {:exception=>RuntimeError, :message=>"Could not fetch the configuration, message: Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: [\"localhost:9200\"] and Kibana hosts: [\"localhost:5601\"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/agent.rb:149:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:93:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:362:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
[INFO ] 2018-09-23 14:44:54.589 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
yml's
kibana:
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://localhost:9200"
elasticsearch:
network.host: localhost
http.port: 9200
I know esearch's running:
curl http://localhost:9200
{
"name" : "97Gu603",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "vcIPJDjZTJScJDFqctebcw",
"version" : {
"number" : "6.4.1",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "e36acdb",
"build_date" : "2018-09-13T22:18:07.696808Z",
"build_snapshot" : false,
"lucene_version" : "7.4.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
and logstash:
bin/logstash -e 'input { stdin { } } output { stdout {} }'
hello world
{
"message" => "hello world",
"@version" => "1",
"host" => "elk",
"@timestamp" => 2018-09-22T08:22:38.284Z
And Kibana is accessible via browser on the host machine.
I've tried searching the various errors but haven't found any fixes.
All help appreciated.
