Unable to ingest events from ArcSight Smart Connector to Logstash ArcSight Module

We have a 5 node ES (version 5.6.0) cluster running on RHEL 7. In addition, 3 of the nodes have Logstash and the remaining 2 nodes have Kibana. The X-pack plugin for Elasticsearch, Logstash and Kibana has been installed and the ArcSight Module has been configured in the logstash.yml. Elasticsearch, Logstash and Kibana seem to startup fine; however, we are not seeing any events forwarded from ArcSight being ingested by Logstash. As a result, Logstash was restarted in debug mode and produced the following error:

[2017-10-05T13:43:48,799][DEBUG][logstash.config.modulescommon] Specified modules {:modules_array=>"[{"name"=>"arcsight", "var.inputs"=>"smartconnector", "var.input.smartconnector.port"=>"1514", "var.elasticsearch.hosts"=>["dw01.dcpds.cpms.osd.mil:9200", "dw02.dcpds.cpms.osd.mil:9200", "dw03.dcpds.cpms.osd.mil:9200"], "var.elasticsearch.username"=>"admin", "var.elasticsearch.password"=>"", "var.elasticsearch.ssl.enabled"=>"true", "var.elasticsearch.ssl.verification_mode"=>"disable", "var.elasticsearch.ssl.certificate_authority"=>"/etc/pki/tls/certs/ca-bundle.crt", "var.elasticsearch.ssl.certificate"=>"/etc/pki/tls/certs/localhost.crt", "var.elasticsearch.ssl.key"=>"/etc/pki/tls/private/logstash.key", "var.kibana.username"=>"kibanaserver", "var.kibana.password"=>"", "var.kibana.ssl.enabled"=>"true", "var.kibana.ssl.verification_mode"=>"disable", "var.kibana.ssl.certificate_authority"=>"/etc/pki/tls/certs/ca-bundle.crt", "var.kibana.ssl.certificate"=>"/etc/pki/tls/certs/localhost.crt", "var.kibana.ssl.key"=>"/etc/pki/tls/private/logstash.key"}]"}
[2017-10-05T13:43:48,875][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::ConfigLoadingError: Failed to parse the module configuration: [Invalid value for :verify. Valid values are (:all, :browser, :default)]>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:614:in ssl_socket_factory_from_options'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:394:inpool_builder'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:402:in pool'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.1-java/lib/manticore/client.rb:208:ininitialize'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-5.0.4/lib/elasticsearch/transport/transport/http/manticore.rb:58:in build_client'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-5.0.4/lib/elasticsearch/transport/transport/http/manticore.rb:49:ininitialize'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-5.0.4/lib/elasticsearch/transport/client.rb:121:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-5.0.4/lib/elasticsearch/transport.rb:26:innew'", "/usr/share/logstash/logstash-core/lib/logstash/elasticsearch_client.rb:47:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/elasticsearch_client.rb:21:inbuild'", "/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:72:in pipeline_configs'", "org/jruby/RubyArray.java:1613:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:56:in pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:272:inexecute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:204:inrun'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in(root)'"]}

Here's the command used to start Logstash with debug enabled and to setup the ArcSight module:
/usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/ --log.level=debug --setup -M "arcsight.var.inputs=smartconnector" -M "arcsight.var.input.smartconnector.port=1514" -M "arcsight.var.elasticsearch.hosts=[dw01.dcpds.cpms.osd.mil:9200,dw02.dcpds.cpms.osd.mil:9200,dw03.dcpds.cpms.osd.mil:9200]" -M "arcsight.var.elasticsearch.username=admin" -M "arcsight.var.elasticsearch.password=" -M "arcsight.var.elasticsearch.ssl.enabled=true" -M "arcsight.var.elasticsearch.ssl.verification_mode=disable" -M "arcsight.var.elasticsearch.ssl.certificate_authority=/etc/pki/tls/certs/ca-bundle.crt" -M "arcsight.var.elasticsearch.ssl.certificate=/etc/pki/tls/certs/localhost.crt" -M "arcsight.var.elasticsearch.ssl.key=/etc/pki/tls/private/logstash.key" -M "arcsight.var.kibana.username=kibanaserver" -M "arcsight.var.kibana.password=" -M "arcsight.var.kibana.ssl.enabled=true" -M "arcsight.var.kibana.ssl.verification_mode=disable" -M "arcsight.var.kibana.ssl.certificate_authority=/etc/pki/tls/certs/ca-bundle.crt" -M "arcsight.var.kibana.ssl.certificate=/etc/pki/tls/certs/localhost.crt" -M "arcsight.var.kibana.ssl.key=/etc/pki/tls/private/logstash.key"

Here's the contents of the logstash.yml file:
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d
modules:

  • name: arcsight
    var.inputs: "smartconnector"
    var.input.smartconnector.port: "1514"
    var.elasticsearch.hosts: [ "dw01.dcpds.cpms.osd.mil:9200", "dw02.dcpds.cpms.osd.mil:9200", "dw03.dcpds.cpms.osd.mil:9200" ]
    var.elasticsearch.username: "admin"
    var.elasticsearch.password: ""
    var.elasticsearch.ssl.enabled: "true"
    var.elasticsearch.ssl.verification_mode: "disable"
    var.elasticsearch.ssl.certificate_authority: "/etc/pki/tls/certs/ca-bundle.crt"
    var.elasticsearch.ssl.certificate: "/etc/pki/tls/certs/localhost.crt"
    var.elasticsearch.ssl.key: "/etc/pki/tls/private/logstash.key"
    var.kibana.username: "kibanaserver"
    var.kibana.password: ""
    var.kibana.ssl.enabled: "true"
    var.kibana.ssl.verification_mode: "disable"
    var.kibana.ssl.certificate_authority: "/etc/pki/tls/certs/ca-bundle.crt"
    var.kibana.ssl.certificate: "/etc/pki/tls/certs/localhost.crt"
    var.kibana.ssl.key: "/etc/pki/tls/private/logstash.key"
    path.logs: /var/log/logstash
    xpack.monitoring.enabled: false

Here's the contents of the Logstash config file:
output {
file {
path => "/var/lib/elasticsearch/temp/logstash.log"
}
}

If you need any more information, please let me know. Appreciate any help you can provide.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.