Config Logstash with https to Elasticsearch cluster

Elastic Stack Logstash
1

Error Logs in Logstash

[2020-09-23T12:24:31,929][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://palelasticsearch01mgt:9200/", :error_type=>LogStash::Outputs::ElasticSearc
h::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://palelasticsearch01mgt:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderExcep
tion: unable to find valid certification path to requested target"}
[2020-09-23T12:24:31,950][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://palelasticsearch02mgt:9200/", :error_type=>LogStash::Outputs::ElasticSearc
h::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://palelasticsearch02mgt:9200/][Manticore::ClientProtocolException] **PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderExcep**
**tion: unable to find valid certification path to requested target"}**
[2020-09-23T12:24:31,974][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://palelasticsearch03mgt:9200/", :error_type=>LogStash::Outputs::ElasticSearc
h::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://palelasticsearch03mgt:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderExcep
tion: unable to find valid certification path to requested target"}
[2020-09-23T12:24:32,870][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://palelasticsearch01mgt:9200/", :error_type=>LogStash::Outputs::ElasticSearch
::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://palelasticsearch01mgt:9200/][Manticore::ClientProtocolException] palelasticsearch01mgt:9200 failed to respond"}
[2020-09-23T12:24:32,874][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://palelasticsearch02mgt:9200/", :error_type=>LogStash::Outputs::ElasticSearch
::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://palelasticsearch02mgt:9200/][Manticore::ClientProtocolException] palelasticsearch02mgt:9200 failed to respond"}
[2020-09-23T12:24:32,877][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://palelasticsearch03mgt:9200/", :error_type=>LogStash::Outputs::ElasticSearch
::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://palelasticsearch03mgt:9200/][Manticore::ClientProtocolException] palelasticsearch03mgt:9200 failed to respond"}

Logstash Config:

output {
if [fields][type] == "glassfish_log" or [fields][type] == "jars_log" {
elasticsearch {
hosts => ["palelasticsearch01mgt:9200","palelasticsearch02mgt:9200","palelasticsearch03mgt:9200"]
index => "app-filebeat"
template_name => "app-filebeat"
template_overwrite => "true"
#ilm_enabled => false
#ilm_rollover_alias => "app-filebeat"
#ilm_pattern => "{now/d}-000001"
#ilm_policy => "app-filebeat"
#truststore => "/etc/logstash/certs/http.p12"
#keystore => "/etc/logstash/certs/http.p12"
#keystore_password => ""
#truststore_password => ""
ssl => true
ssl_certificate_verification => false
user => 'logstash_writer'
password => 'xxxx'
}
}
else if [fields][type] == "nginx_access_log" or [fields][type] == "nginx_error_log" {
elasticsearch {
hosts => ["palelasticsearch01mgt:9200","palelasticsearch02mgt:9200","palelasticsearch03mgt:9200"]
index => "proxy-filebeat"
template_name => "proxy-filebeat"
template_overwrite => "true"
#ilm_enabled => false
#ilm_rollover_alias => "proxy-filebeat"
#ilm_pattern => "{now/d}-000001"
#ilm_policy => "proxy-filebeat"
#truststore => "/etc/logstash/certs/pallogstash01mgt.p12"
#keystore => "/etc/logstash/certs/pallogstash01mgt.p12"
#keystore_password => ""
#truststore_password => ""
ssl => true
ssl_certificate_verification => false
user => 'logstash_writer'
password => 'xxxx'
}
}
}

Elastic Search Logs:

[2020-09-23T05:30:31,456][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [palelasticsearch01mgt] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/192.168.24.60:9200, remote
Address=/192.168.24.67:56596}
[2020-09-23T05:30:34,652][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [palelasticsearch01mgt] http client did not trust this server's certificate, closing connection Netty4HttpChannel{localAddress=/192.168.24.60:9200, remote
Address=/192.168.24.67:56602}
[2020-09-23T05:30:36,465][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [palelasticsearch01mgt] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/192.168.24.60:9200, remote
Address=/192.168.24.67:56608}
[2020-09-23T05:30:39,731][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [palelasticsearch01mgt] http client did not trust this server's certificate, closing connection Netty4HttpChannel{localAddress=/192.168.24.60:9200, remote
Address=/192.168.24.67:56614}
[2020-09-23T05:30:41,474][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [palelasticsearch01mgt] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/192.168.24.60:9200, remote
Address=/192.168.24.67:56620}
[2020-09-23T05:30:44,804][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [palelasticsearch01mgt] http client did not trust this server's certificate, closing connection Netty4HttpChannel{localAddress=/192.168.24.60:9200, remote
Address=/192.168.24.67:56626}
[2020-09-23T05:30:46,484][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [palelasticsearch01mgt] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/192.168.24.60:9200, remote
Address=/192.168.24.67:56632}

I have changed many configs in logstash but didn't work, the cert in logstash config has come from Elasticsearch CA (PKSC12 elastic-stack-ca.p12 ) with truststore config or CACERT ( export pkcs12 to pem).

I don't know how to config logstash correctly. please help me.
Additionally, logstash is monitored by elasticsearch already. just unable to push logs to Elasticsearch.

2

In your logstash config, cacert field is missing and hosts are not https.

output {
if [fields][type] == "glassfish_log" or [fields][type] == "jars_log" {
  elasticsearch {
    hosts => ["https://palelasticsearch01mgt:9200","https://palelasticsearch02mgt:9200","https://palelasticsearch03mgt:9200"]
    index => "app-filebeat"
    template_name => "app-filebeat"
    template_overwrite => "true"
    #ilm_enabled => false
    #ilm_rollover_alias => "app-filebeat"
    #ilm_pattern => "{now/d}-000001"
    #ilm_policy => "app-filebeat"
    #truststore => "/etc/logstash/certs/http.p12"
    #keystore => "/etc/logstash/certs/http.p12"
    #keystore_password => ""
    #truststore_password => ""
    ssl => true
    ssl_certificate_verification => false
    user => 'logstash_writer'
    password => 'xxxx'
    cacert => "YOUR_CACERT"
  }
}

You can refer to below for reference.

Section [6-4] Create and configure conf.d/example.conf

Hope this could help you.

3

Your answer is almost true :smiley:
because of my cert is not valid so the way that I did was not work. I updated the Cert and change the hosts to HTTPS and it's working now.
Thanks for your answer.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.