Logstash Arcsight Module Implementation Issues

Completed installation of the ELK stack version 8.4 - single node installation
Used a test pipeline to confirm comms between Elasticsearch and Logstash on local machine on the commandline interface.

So, I tried implementing the ArcSight module by both using the Logstash yml file and through the command-line using both sets of config:

Command-line

./logstash --modules arcsight --setup -M "arcsight.var.input.smartconnector.port=5000" -M "arcsight.var.elasticsearch.hosts=https://localhost:9200" -M "arcsight.var.elasticsearch.username=elastic" -M "arcsight.var.elasticsearch.password=PASSWORD" -M "arcsight.var.kibana.hosts=localhost:5601" -M "arcsight.var.kibana.username=elastic" -M "arcsight.var.kibana.password=PASSWORD" -M "arcsight.var.kibana.ssl.enabled=false" -M "arcsight.var.elasticsearch.ssl.certificate=/etc/logstash/certs/http_ca.crt"

Logstash.yml

- name: arcsight arcsight.var.input.smartconnector.port: 5000 arcsight.var.elasticsearch.hosts: ["https://localhost:9200"] arcsight.var.elasticsearch.username: "elastic" arcsight.var.elasticsearch.password: "PASSWORD" arcsight.var.kibana.host: "localhost:5601" arcsight.var.kibana.username: "elastic" arcsight.var.kibana.password: "PASSWORD" arcsight.var.elasticsearch.ssl.certificate: "/etc/logstash/certs/http_ca.crt"

Error Messages after running both command-line and logastash.yml configurations

`
Using bundled JDK: /usr/share/logstash/jdk

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults

Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console

[WARN ] 2022-11-02 10:24:29.443 [main] runner - NOTICE: Running Logstash as superuser is not recommended and won't be allowed in the future. Set 'allow_superuser' to 'false' to avoid startup errors in future releases.

[INFO ] 2022-11-02 10:24:29.453 [main] runner - Starting Logstash {"logstash.version"=>"8.4.3", "jruby.version"=>"jruby 9.3.8.0 (2.6.8) 2022-09-13 98d69c9461 OpenJDK 64-Bit Server VM 17.0.4+8 on 17.0.4+8 +indy +jit [x86_64-linux]"}

[INFO ] 2022-11-02 10:24:29.455 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]

[WARN ] 2022-11-02 10:24:29.643 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified

[INFO ] 2022-11-02 10:24:30.355 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}

[INFO ] 2022-11-02 10:24:30.660 [Agent thread] licensereader - Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[https://elastic:xxxxxx@localhost:9200/]}}

[INFO ] 2022-11-02 10:24:30.832 [Agent thread] licensereader - Failed to perform request {:message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target>}

[WARN ] 2022-11-02 10:24:30.835 [Agent thread] licensereader - Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@localhost:9200/", :exception=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [[https://localhost:9200/]Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}

[INFO ] 2022-11-02 10:24:30.874 [Agent thread] licensereader - Failed to perform request {:message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target>}

[WARN ] 2022-11-02 10:24:30.878 [Agent thread] licensereader - Marking url as dead. Last error: [LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [[https://localhost:9200/_xpack]Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target {:url=>https://elastic:xxxxxx@localhost:9200/, :error_message=>"Elasticsearch Unreachable: [[https://localhost:9200/_xpack]Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :error_class=>"LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError"}

[WARN ] 2022-11-02 10:24:30.886 [Agent thread] licensereader - Attempt to validate Elasticsearch license failed. Sleeping for 0.02 {:fail_count=>1, :exception=>"Elasticsearch Unreachable: [[https://localhost:9200/_xpack]Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}

[ERROR] 2022-11-02 10:24:30.909 [Agent thread] licensereader - Unable to retrieve license information from license server {:message=>"No Available connections"}

[ERROR] 2022-11-02 10:24:30.932 [Agent thread] modulelicensechecker - Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.

[WARN ] 2022-11-02 10:24:30.932 [Agent thread] modulescommon - The arcsight module is not enabled. Please check the logs for additional information.

[ERROR] 2022-11-02 10:24:30.936 [Agent thread] sourceloader - No configuration found in the configured sources .

[INFO ] 2022-11-02 10:24:31.001 [LogStash::Runner] runner - Logstash shut down.
`

Just to confirm I was able to run a test pipeline from the /etc/logstash/conf.d without any issues so I can confirm Logstash and ES are able to contact each other securely.

Below is a confirmation that I could curl ES on HTTPS which would confirm ES is running despite the errors indicating that the instance is unreachable
curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200 Enter host password for user 'elastic': { "name" : "HOSTNAME", "cluster_name" : "elasticsearch", "cluster_uuid" : "Hpk7yGniS1S8aASBpe9ebA", "version" : { "number" : "8.4.3", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "42f05b9372a9a4a470db3b52817899b99a76ee73", "build_date" : "2022-10-04T07:17:24.662462378Z", "build_snapshot" : false, "lucene_version" : "9.3.0", "minimum_wire_compatibility_version" : "7.17.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "You Know, for Search" }

Any help on how to resolve this will be highly appreciated!

Up at the top it says:

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults

Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console

When this happened to me, I resolved it by including this in my command to start logstash:

--path.settings /etc/logstash

Full command example for starting logstash:

sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/config.conf

See if including "--path.settings /etc/logstash" in your command to start Logstash just like the above example resolves this for you too.

Cheers,
Ian

Thanks for your quick response!

I've tried exactly that by running below from the logstash installing directory:

./logstash --modules arcsight --setup --path.settings /etc/logstash

The error message I get is:
Unable to reach license information from license server

Also:

Failed to fetch X-pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster