Config Logstash with https to Elasticsearch cluster

Error Logs in Logstash

[2020-09-23T12:24:31,929][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://palelasticsearch01mgt:9200/", :error_type=>LogStash::Outputs::ElasticSearc
h::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://palelasticsearch01mgt:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderExcep
tion: unable to find valid certification path to requested target"}
[2020-09-23T12:24:31,950][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://palelasticsearch02mgt:9200/", :error_type=>LogStash::Outputs::ElasticSearc
h::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://palelasticsearch02mgt:9200/][Manticore::ClientProtocolException] **PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderExcep**
**tion: unable to find valid certification path to requested target"}**
[2020-09-23T12:24:31,974][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://palelasticsearch03mgt:9200/", :error_type=>LogStash::Outputs::ElasticSearc
h::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://palelasticsearch03mgt:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderExcep
tion: unable to find valid certification path to requested target"}
[2020-09-23T12:24:32,870][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://palelasticsearch01mgt:9200/", :error_type=>LogStash::Outputs::ElasticSearch
::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://palelasticsearch01mgt:9200/][Manticore::ClientProtocolException] palelasticsearch01mgt:9200 failed to respond"}
[2020-09-23T12:24:32,874][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://palelasticsearch02mgt:9200/", :error_type=>LogStash::Outputs::ElasticSearch
::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://palelasticsearch02mgt:9200/][Manticore::ClientProtocolException] palelasticsearch02mgt:9200 failed to respond"}
[2020-09-23T12:24:32,877][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://palelasticsearch03mgt:9200/", :error_type=>LogStash::Outputs::ElasticSearch
::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://palelasticsearch03mgt:9200/][Manticore::ClientProtocolException] palelasticsearch03mgt:9200 failed to respond"}

Logstash Config:

output {
if [fields][type] == "glassfish_log" or [fields][type] == "jars_log" {
elasticsearch {
hosts => ["palelasticsearch01mgt:9200","palelasticsearch02mgt:9200","palelasticsearch03mgt:9200"]
index => "app-filebeat"
template_name => "app-filebeat"
template_overwrite => "true"
#ilm_enabled => false
#ilm_rollover_alias => "app-filebeat"
#ilm_pattern => "{now/d}-000001"
#ilm_policy => "app-filebeat"
#truststore => "/etc/logstash/certs/http.p12"
#keystore => "/etc/logstash/certs/http.p12"
#keystore_password => ""
#truststore_password => ""
ssl => true
ssl_certificate_verification => false
user => 'logstash_writer'
password => 'xxxx'
}
}
else if [fields][type] == "nginx_access_log" or [fields][type] == "nginx_error_log" {
elasticsearch {
hosts => ["palelasticsearch01mgt:9200","palelasticsearch02mgt:9200","palelasticsearch03mgt:9200"]
index => "proxy-filebeat"
template_name => "proxy-filebeat"
template_overwrite => "true"
#ilm_enabled => false
#ilm_rollover_alias => "proxy-filebeat"
#ilm_pattern => "{now/d}-000001"
#ilm_policy => "proxy-filebeat"
#truststore => "/etc/logstash/certs/pallogstash01mgt.p12"
#keystore => "/etc/logstash/certs/pallogstash01mgt.p12"
#keystore_password => ""
#truststore_password => ""
ssl => true
ssl_certificate_verification => false
user => 'logstash_writer'
password => 'xxxx'
}
}
}

Elastic Search Logs:

[2020-09-23T05:30:31,456][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [palelasticsearch01mgt] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/192.168.24.60:9200, remote
Address=/192.168.24.67:56596}
[2020-09-23T05:30:34,652][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [palelasticsearch01mgt] http client did not trust this server's certificate, closing connection Netty4HttpChannel{localAddress=/192.168.24.60:9200, remote
Address=/192.168.24.67:56602}
[2020-09-23T05:30:36,465][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [palelasticsearch01mgt] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/192.168.24.60:9200, remote
Address=/192.168.24.67:56608}
[2020-09-23T05:30:39,731][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [palelasticsearch01mgt] http client did not trust this server's certificate, closing connection Netty4HttpChannel{localAddress=/192.168.24.60:9200, remote
Address=/192.168.24.67:56614}
[2020-09-23T05:30:41,474][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [palelasticsearch01mgt] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/192.168.24.60:9200, remote
Address=/192.168.24.67:56620}
[2020-09-23T05:30:44,804][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [palelasticsearch01mgt] http client did not trust this server's certificate, closing connection Netty4HttpChannel{localAddress=/192.168.24.60:9200, remote
Address=/192.168.24.67:56626}
[2020-09-23T05:30:46,484][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [palelasticsearch01mgt] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/192.168.24.60:9200, remote
Address=/192.168.24.67:56632}

I have changed many configs in logstash but didn't work, the cert in logstash config has come from elastic search CA (PKSC12 elastic-stack-ca.p12 ) with truststore config or CACERT ( export pkcs12 to pem).

I don't know how to config logstash correctly. please help me.
Additionally, logstash is monitored by elasticsearch already. just unable to push logs to elastic search.

In your logstash config, cacert field is missing and hosts are not https.

output {
if [fields][type] == "glassfish_log" or [fields][type] == "jars_log" {
  elasticsearch {
    hosts => ["https://palelasticsearch01mgt:9200","https://palelasticsearch02mgt:9200","https://palelasticsearch03mgt:9200"]
    index => "app-filebeat"
    template_name => "app-filebeat"
    template_overwrite => "true"
    #ilm_enabled => false
    #ilm_rollover_alias => "app-filebeat"
    #ilm_pattern => "{now/d}-000001"
    #ilm_policy => "app-filebeat"
    #truststore => "/etc/logstash/certs/http.p12"
    #keystore => "/etc/logstash/certs/http.p12"
    #keystore_password => ""
    #truststore_password => ""
    ssl => true
    ssl_certificate_verification => false
    user => 'logstash_writer'
    password => 'xxxx'
    cacert => "YOUR_CACERT"
  }
}

You can refer to below for reference.

Section [6-4] Create and configure conf.d/example.conf

Hope this could help you.

Your answer is almost true :smiley:
because of my cert is not valid so the way that I did was not work. I updated the Cert and change the hosts to HTTPS and it's working now.
Thanks for your answer.