【Logstash】The output configuration of logstash cannot connect to the elasticsearch

Elastic Stack Logstash
1

I build a single-node of elasticsearch on GCP and a logstash on the local side. I want to connect the output configuration of logstash to elasticsearch.

Info:
Elasticsearch、Kibana、Logstash: 8.6.1.
I set up an extenal IP on GCP via static.

1. Create CA
1-1. Create ssl.yml

vi ssl.yml

instances:
  - name: "poc-elk"
    ip:
      - "<GCP_External_IP>"
      - "<GCP_Internal_IP>"
      - "<Local_logstash_IP>"
    dns:
      - "<GCP_hostname>"
      - "<Local_hostname>"

1-2. Create ca

/usr/share/elasticsearch/bin/elasticsearch-certgen --dn 'CN=elk-ca' --days 7300 --keysize 4096 --in /etc/elasticsearch/certs/ssl.yml --out /etc/elasticsearch/certs/ssl.zip

1-3. unzip ssl.zip and the catalog structure.

Archive:  ssl.zip
   creating: ca/
  inflating: ca/ca.crt               
  inflating: ca/ca.key               
   creating: poc-elk/
  inflating: poc-elk/poc-elk.crt     
  inflating: poc-elk/poc-elk.key

2. elasticsearch.yml on GCP

cluster.name: elk 
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  key: /etc/elasticsearch/poc-elk.key
  certificate: /etc/elasticsearch/poc-elk.crt
  certificate_authorities: /etc/elasticsearch/ca.crt
cluster.initial_master_nodes: ["node-1"]
http.host: 0.0.0.0

3. logstash.yml on local side

node.name: lgs
path.data: /var/lib/logstash
path.logs: /var/log/logstash

4. logstash output

less /etc/logstash/conf.d/file-test.conf

input{
    file{
        path => "/etc/logstash/conf.d/file-test.txt"
    }
}
output{
    elasticsearch{
        hosts => ["<GCP_External_IP>:9200"]
        user => "elastic"
        password => "<MyPass>"
        ssl => true
        cacert => "/etc/logstash/certs/ca.crt"
    }
}

Test:

echo -n  "test123" > /etc/logstash/conf.d/file-test.txt

5. Logstash Error messages

less /var/log/logstash/logstash-plain.log

[2023-02-14T17:49:48,450][INFO ][logstash.outputs.elasticsearch][file-test] Failed to perform request {:message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target>}
[2023-02-14T17:49:48,462][WARN ][logstash.outputs.elasticsearch][file-test] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@<GCP_External_IP>:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://<GCP_External_IP>:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
[2023-02-14T17:49:48,749][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>34, "name"=>"[file-test]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.6.0/gems/stud-0.0.23/lib/stud/interval.rb:95:in `sleep'"}, {"thread_id"=>35, "name"=>"[file-test]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.6.0/gems/stud-0.0.23/lib/stud/interval.rb:95:in `sleep'"}]}}
2

[quote="Roy176, post:1, topic:325523"][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}`
[/quote]
You have issue with keys.

  creating: ca/
  inflating: ca/ca.crt               
  inflating: ca/ca.key               
   creating: poc-elk/
  inflating: poc-elk/poc-elk.crt     
  inflating: poc-elk/poc-elk.key

The poc ca files were extracted in a subdirectory.

Should be or move the ca files in /etc/elasticsearch/:

  key: /etc/elasticsearch/poc-elk/poc-elk.key
  certificate: /etc/elasticsearch/poc-elk/poc-elk.crt
  certificate_authorities: /etc/elasticsearch/poc-elk/ca.crt
3

@Rios
Thanks for your reply. The error message is on the local site logstash, so I don't think changing the location of the elasticsearch certificate on GCP is valid.

4

I think the 2nd line of your Logstash error messages may be your problem, "Elasticsearch Unreachable: [https://<GCP_External_IP>:9200/]".

From your Logstash host, can you manually curl to that URL using the "-v" and "-k" options? The "-k" skips certificate checks and assumes they are valid.

So, that should tell you whether it is routing or certificates or something else.