Netflow --setup to ES SSL not working

Elastic Stack Logstash
1

Here is my command line attempt to setup netflow in kibana:

/usr/share/logstash/bin/logstash --modules netflow --setup -M "netflow.var.kibana.host=:5601" -M "netflow.var.kibana.username=elastic" -M "netflow.var.kibana.password=**" -M "netflow.var.input.udp.port=9996" -M "netflow.var.elasticsearch.hosts=https://**:9200" -M "netflow.var.elasticsearch.username=elastic" -M "netflow.var.elasticsearch.password=" -M "netflow.var.elasticsearch.ssl.certificate_authority=/etc/logstash/ca.crt"

Heres is the output:
[WARN ] 2019-09-20 10:01:03.151 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-09-20 10:01:03.882 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.1.1"}
[INFO ] 2019-09-20 10:01:10.663 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] modulescommon - Setting up the netflow module
[ERROR] 2019-09-20 10:01:15.185 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] kibanaclient - Error when executing Kibana client request {:error=>#<Manticore::ClientProtocolException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target>}
[ERROR] 2019-09-20 10:01:16.756 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] kibanaclient - Error when executing Kibana client request {:error=>#<Manticore::ClientProtocolException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target>}
[ERROR] 2019-09-20 10:01:17.962 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] sourceloader - Could not fetch all the sources {:exception=>LogStash::ConfigLoadingError, :message=>"Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: ["https://cloudcontrol-elasticsearch1.office.re.local:9200"] and Kibana hosts: ["cloudcontrol-elasticsearch1.office.re.local:5601"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:108:in block in pipeline_configs'", "org/jruby/RubyArray.java:1792:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:54:in pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/modules.rb:14:inpipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:61:in block in fetch'", "org/jruby/RubyArray.java:2572:incollect'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:60:in fetch'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:148:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:362:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/task.rb:24:in block in initialize'"]} [ERROR] 2019-09-20 10:01:18.552 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - An exception happened when converging configuration {:exception=>RuntimeError, :message=>"Could not fetch the configuration, message: Failed to import module configurations to Elasticsearch and/or Kibana. Module: netflow has Elasticsearch hosts: [\"https://cloudcontrol-elasticsearch1.office.re.local:9200\"] and Kibana hosts: [\"cloudcontrol-elasticsearch1.office.re.local:5601\"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/agent.rb:155:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:362:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
[INFO ] 2019-09-20 10:01:20.196 [LogStash::Runner] runner - Logstash shut down.

Here is the ca validation:
root@cloudcontrol-logstash:~# curl --cacert /etc/logstash/ca.crt -u elastic:***** -XGET 'https://*:9200'
{
"name" : "",
"cluster_name" : "ccc20-basic",
"cluster_uuid" : "l8D7mil9RLyqXEHVkuEQyQ",
"version" : {
"number" : "7.3.1",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "4749ba6",
"build_date" : "2019-08-19T20:19:25.651794Z",
"build_snapshot" : false,
"lucene_version" : "8.1.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

2

I needed to add kibana ssl cert and logstash is running now.

It created the dashboards but no netflow index:

/usr/share/logstash/bin/logstash --modules netflow --setup -M "netflow.var.kibana.host=************************:5601" -M "netflow.var.kibana.ssl.enabled=true" -M "netflow.var.kibana.ssl.enabled=true" -M "netflow.var.kibana.scheme=https" -M "netflow.var.kibana.ssl.certificate_authority=/etc/logstash/ca.crt" -M "netflow.var.kibana.username=elastic" -M "netflow.var.kibana.password=************************" -M "netflow.var.input.udp.port=9996" -M "netflow.var.elasticsearch.hosts=https://************************:9200" -M "netflow.var.elasticsearch.username=elastic" -M "netflow.var.elasticsearch.password=************************" -M "netflow.var.elasticsearch.ssl.certificate_authority=/etc/logstash/ca.crt"

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-09-20 11:18:54.494 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-09-20 11:18:54.517 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.1.1"}
[INFO ] 2019-09-20 11:18:59.250 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] modulescommon - Setting up the netflow module
[INFO ] 2019-09-20 11:20:23.956 [[module-netflow]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[https://elastic:xxxxxx@cloudcontrol-elasticsearch1.office.re.local:9200/]}}
[WARN ] 2019-09-20 11:20:24.306 [[module-netflow]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://elastic:xxxxxx@cloudcontrol-elasticsearch1.office.re.local:9200/"}
[INFO ] 2019-09-20 11:20:24.338 [[module-netflow]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2019-09-20 11:20:24.436 [[module-netflow]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2019-09-20 11:20:24.476 [[module-netflow]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://cloudcontrol-elasticsearch1.office.re.local:9200"]}
[INFO ] 2019-09-20 11:20:26.124 [[module-netflow]-pipeline-manager] geoip - Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.1-java/vendor/GeoLite2-ASN.mmdb"}
[INFO ] 2019-09-20 11:20:26.559 [[module-netflow]-pipeline-manager] geoip - Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.1-java/vendor/GeoLite2-City.mmdb"}
[INFO ] 2019-09-20 11:20:26.563 [[module-netflow]-pipeline-manager] geoip - Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.1-java/vendor/GeoLite2-City.mmdb"}
[INFO ] 2019-09-20 11:20:27.632 [[module-netflow]-pipeline-manager] geoip - Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.1-java/vendor/GeoLite2-ASN.mmdb"}
[INFO ] 2019-09-20 11:20:27.642 [[module-netflow]-pipeline-manager] geoip - Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.1-java/vendor/GeoLite2-ASN.mmdb"}
[INFO ] 2019-09-20 11:20:27.643 [[module-netflow]-pipeline-manager] geoip - Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.1-java/vendor/GeoLite2-City.mmdb"}
[INFO ] 2019-09-20 11:20:28.607 [[module-netflow]-pipeline-manager] geoip - Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.1-java/vendor/GeoLite2-City.mmdb"}
[INFO ] 2019-09-20 11:20:29.564 [[module-netflow]-pipeline-manager] geoip - Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.1-java/vendor/GeoLite2-ASN.mmdb"}
[INFO ] 2019-09-20 11:20:29.609 [[module-netflow]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"module-netflow", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, :thread=>"#<Thread:0x68704d94 run>"}
[INFO ] 2019-09-20 11:20:29.691 [[module-netflow]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"module-netflow"}
[INFO ] 2019-09-20 11:20:30.596 [[module-netflow]<udp] udp - Starting UDP listener {:address=>"0.0.0.0:9996"}
[INFO ] 2019-09-20 11:20:31.085 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:"module-netflow"], :non_running_pipelines=>}
[INFO ] 2019-09-20 11:20:31.106 [[module-netflow]<udp] udp - UDP listener started {:address=>"0.0.0.0:9996", :receive_buffer_bytes=>"212992", :queue_size=>"2000"}
[INFO ] 2019-09-20 11:20:33.765 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9604}

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.