We have a 5 node ES (version 5.6.0) cluster running on RHEL 7. In addition, 3 of the nodes have Logstash and the remaining 2 nodes have Kibana. The X-pack plugin for Elasticsearch, Logstash and Kibana has been installed and the ArcSight Module has been configured in the logstash.yml. Elasticsearch, Logstash and Kibana seem to startup fine; however, we are not seeing any events forwarded from ArcSight being ingested by Logstash. As a result, Logstash was restarted in debug mode and produced the following error:
[2017-10-05T13:43:48,799][DEBUG][logstash.config.modulescommon] Specified modules {:modules_array=>"[{"name"=>"arcsight", "var.inputs"=>"smartconnector", "var.input.smartconnector.port"=>"1514", "var.elasticsearch.hosts"=>["dw01.dcpds.cpms.osd.mil:9200", "dw02.dcpds.cpms.osd.mil:9200", "dw03.dcpds.cpms.osd.mil:9200"], "var.elasticsearch.username"=>"admin", "var.elasticsearch.password"=>"", "var.elasticsearch.ssl.enabled"=>"true", "var.elasticsearch.ssl.verification_mode"=>"disable", "var.elasticsearch.ssl.certificate_authority"=>"/etc/pki/tls/certs/ca-bundle.crt", "var.elasticsearch.ssl.certificate"=>"/etc/pki/tls/certs/localhost.crt", "var.elasticsearch.ssl.key"=>"/etc/pki/tls/private/logstash.key", "var.kibana.username"=>"kibanaserver", "var.kibana.password"=>"", "var.kibana.ssl.enabled"=>"true", "var.kibana.ssl.verification_mode"=>"disable", "var.kibana.ssl.certificate_authority"=>"/etc/pki/tls/certs/ca-bundle.crt", "var.kibana.ssl.certificate"=>"/etc/pki/tls/certs/localhost.crt", "var.kibana.ssl.key"=>"/etc/pki/tls/private/logstash.key"}]"}
[2017-10-05T13:43:48,875][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::ConfigLoadingError: Failed to parse the module configuration: [Invalid value for :verify. Valid values are (:all, :browser, :default)]>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:614:in ssl_socket_factory_from_options'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:394:inpool_builder'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:402:in pool'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:208:ininitialize'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-5.0.4/lib/elasticsearch/transport/transport/http/manticore.rb:58:in build_client'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-5.0.4/lib/elasticsearch/transport/transport/http/manticore.rb:49:ininitialize'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-5.0.4/lib/elasticsearch/transport/client.rb:121:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-5.0.4/lib/elasticsearch/transport.rb:26:innew'", "/usr/share/logstash/logstash-core/lib/logstash/elasticsearch_client.rb:47:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/elasticsearch_client.rb:21:inbuild'", "/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:72:in pipeline_configs'", "org/jruby/RubyArray.java:1613:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:56:in pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:272:inexecute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:204:inrun'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in(root)'"]}
Here's the command used to start Logstash with debug enabled and to setup the ArcSight module:
/usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/ --log.level=debug --setup -M "arcsight.var.inputs=smartconnector" -M "arcsight.var.input.smartconnector.port=1514" -M "arcsight.var.elasticsearch.hosts=[dw01.dcpds.cpms.osd.mil:9200,dw02.dcpds.cpms.osd.mil:9200,dw03.dcpds.cpms.osd.mil:9200]" -M "arcsight.var.elasticsearch.username=admin" -M "arcsight.var.elasticsearch.password=" -M "arcsight.var.elasticsearch.ssl.enabled=true" -M "arcsight.var.elasticsearch.ssl.verification_mode=disable" -M "arcsight.var.elasticsearch.ssl.certificate_authority=/etc/pki/tls/certs/ca-bundle.crt" -M "arcsight.var.elasticsearch.ssl.certificate=/etc/pki/tls/certs/localhost.crt" -M "arcsight.var.elasticsearch.ssl.key=/etc/pki/tls/private/logstash.key" -M "arcsight.var.kibana.username=kibanaserver" -M "arcsight.var.kibana.password=" -M "arcsight.var.kibana.ssl.enabled=true" -M "arcsight.var.kibana.ssl.verification_mode=disable" -M "arcsight.var.kibana.ssl.certificate_authority=/etc/pki/tls/certs/ca-bundle.crt" -M "arcsight.var.kibana.ssl.certificate=/etc/pki/tls/certs/localhost.crt" -M "arcsight.var.kibana.ssl.key=/etc/pki/tls/private/logstash.key"
Here's the contents of the logstash.yml file:
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d
modules:
- name: arcsight
var.inputs: "smartconnector"
var.input.smartconnector.port: "1514"
var.elasticsearch.hosts: [ "dw01.dcpds.cpms.osd.mil:9200", "dw02.dcpds.cpms.osd.mil:9200", "dw03.dcpds.cpms.osd.mil:9200" ]
var.elasticsearch.username: "admin"
var.elasticsearch.password: ""
var.elasticsearch.ssl.enabled: "true"
var.elasticsearch.ssl.verification_mode: "disable"
var.elasticsearch.ssl.certificate_authority: "/etc/pki/tls/certs/ca-bundle.crt"
var.elasticsearch.ssl.certificate: "/etc/pki/tls/certs/localhost.crt"
var.elasticsearch.ssl.key: "/etc/pki/tls/private/logstash.key"
var.kibana.username: "kibanaserver"
var.kibana.password: ""
var.kibana.ssl.enabled: "true"
var.kibana.ssl.verification_mode: "disable"
var.kibana.ssl.certificate_authority: "/etc/pki/tls/certs/ca-bundle.crt"
var.kibana.ssl.certificate: "/etc/pki/tls/certs/localhost.crt"
var.kibana.ssl.key: "/etc/pki/tls/private/logstash.key"
path.logs: /var/log/logstash
xpack.monitoring.enabled: false
Here's the contents of the Logstash config file:
output {
file {
path => "/var/lib/elasticsearch/temp/logstash.log"
}
}
If you need any more information, please let me know. Appreciate any help you can provide.