My ELK stack got 3 ES nodes and 2 logstash nodes. Kibana is installed on one of the Logstash nodes. I was able to generate CA and all certificates. Distributed the certificates to all nodes.Confirmed Elasticsearch nodes are joined to the Elastic cluster.
And ES nodes are able to access via https.
My issue is with configuring SSL/TLS for the logstash config and output. I have followed instruction on Configure SSL/TLS for the Logstash output | Fleet and Elastic Agent Guide [8.6] | Elastic document.
I created CA --pem zip file from the ES node-1 and copied that ca.crt and ca.key folder to the logstash server to generate client and Logstash SSL certificate as mentioned in the document.
filebeatlogstash.conf :-
input {
beats {
port => 5046
host => "30.21.332.564"
ssl => true
ssl_certificate_authorities => [ "C:/elastic/elasticsearch-8.6.2/elastic-stack-ca/ca/ca.crt" ]
ssl_certificate => "C:/elastic/logstash-8.6.2/config/certs/logstash.crt"
ssl_key => "C:/elastic/logstash-8.6.2/config/certs/logstash.pkcs8.key"
#ssl_verify_mode => "force_peer"
tags => ["filebeat"]
}
}
output{
#stdout { codec => rubydebug }
if "filebeat" in [tags] {
elasticsearch {
hosts => ["https://30.21.332.566:9200", "https://30.21.332.567:9200", "https://30.21.332.568:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
action => "create"
user => elastic
password => "*******"
ssl => true
cacert => "C:/elastic/elasticsearch-8.6.2/elastic-stack-ca/ca/ca.crt"
}
}
}
When I start the logstash I am getting below error : To start Logstash service, i use the following command : ./bin/logstash -f path to the "filebeatlogstash.conf" file.
2023-09-05T19:32:37,363][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-09-05T19:32:38,645][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-09-05T19:32:39,109][INFO ][org.reflections.Reflections] Reflections took 143 ms to scan 1 urls, producing 127 keys and 444 values
[2023-09-05T19:32:39,508][INFO ][logstash.javapipeline ] Pipeline `mainlogstash` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2023-09-05T19:32:39,518][INFO ][logstash.outputs.elasticsearch][mainlogstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://30.21.332.566:9200", "https://30.21.332.567:9200", "https://30.21.332.568:9200"]}
[2023-09-05T19:32:39,826][INFO ][logstash.outputs.elasticsearch][mainlogstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@30.21.332.566:9200/, https://elastic:xxxxxx@30.21.332.567:9200/, https://elastic:xxxxxx@30.21.332.568:9200/]}}
[2023-09-05T19:32:40,041][INFO ][logstash.outputs.elasticsearch][mainlogstash] Failed to perform request {:message=>"PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors>}
[2023-09-05T19:32:40,044][WARN ][logstash.outputs.elasticsearch][mainlogstash] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@30.21.332.566:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://30.21.332.566:9200/][Manticore::ClientProtocolException] PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors"}
[2023-09-05T19:32:40,092][INFO ][logstash.outputs.elasticsearch][mainlogstash] Failed to perform request {:message=>"PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors>}
[2023-09-05T19:32:40,096][WARN ][logstash.outputs.elasticsearch][mainlogstash] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@30.21.332.567:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://30.21.332.567:9200/][Manticore::ClientProtocolException] PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors"}
[2023-09-05T19:32:40,120][INFO ][logstash.outputs.elasticsearch][mainlogstash] Failed to perform request {:message=>"PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors", :exception=>Manticore::ClientProtocolException, :cause=>#<Java::JavaxNetSsl::SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors>}
[2023-09-05T19:32:40,122][WARN ][logstash.outputs.elasticsearch][mainlogstash] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@30.21.332.568:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://30.21.332.568:9200/][Manticore::ClientProtocolException] PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors"}
[2023-09-05T19:32:40,130][INFO ][logstash.outputs.elasticsearch][mainlogstash] Not eligible for data streams because config contains one or more settings that are not compatible with data streams: {"index"=>"%{[@metadata][beat]}-%{+YYYY.MM.dd}"}
2023-09-05T19:32:45,189][WARN ][logstash.outputs.elasticsearch][mainlogstash] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@30.21.332.568:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://30.21.332.568:9200/][Manticore::ClientProtocolException] PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors"}
[2023-09-05T19:32:45,465][INFO ][org.logstash.beats.BeatsHandler][mainlogstash][3b7f677414e958436012b4fcf95d13fd9514c2b6755eff61252bae0be79e73cd] [local: 30.21.332.564:5046, remote: 10.112.33.2:51137] Handling exception: io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 325700000800324300028e18785eec9dd992dc4857c70514fbbeef20c43233a02a94da55579fede98f9809dbe370b799083e7fe190abd4ddc2b5f448aa9931137e00b8011e051e00ee20d86e20e07988a34c2da54aa952aa544ae51edfd8eeaaae9232cff99dff392795a97f2a49d20f48d2e4dfbf51be9384eb204efcf59d3257744d37a69a37d5ac2b64ce0d776eba33647a7fa1a8ca77d641e22ffdc457e6df28af033f51e6ca75b80a5e077ea2a84af2f62e50e6caabe576a1a8ca97411487db8d3257dc993dd39577aab2dade28f36f94edf5751c24ca1cd9a6a6a61f009f77e727b7ca5c79347ff9f259b4bd89fcf5c77ee2bf7cf934f83a5c86fecb970f36feea6d122ee2972f1f6f6fe2fc95d9437ff1e626daee36cbcfb7d19b20ba0ca22fc34530bbdcdddd6da3e4f1f666967c9d28efdea9ca3a8863ff06aeb2b84d5973e7963e47b66a204746a6a7dba669999afcc9d3ef7e26cbb2a5c97b7f2e936d142ce5bb68bb0896bb2898cb976fe324587fba7d1d5f7c1d2c7649b8ddbcd884c9ec2259bdc23f091efac9e2569eca41f606f93adc84f16db094e3dd6211c4f1f56eb57a2b5f6f23f9d3ed6bf9f96e235f26c19d1c055fec823891c3a56ce9baa9b9d64cfede879bdd6af5d1f7653236b38b38ac1d057c01e17633fb74fbfa72711b2c77ab7073337bbc5df8abcb8b17f1ecf28bd5b3ec66f6ff7719ec1455b90e83d53286395aece264bb0ea2571b7f0d83f83488efa2208eb74a3ab9af88053cfde401f941bc7b4d7e5633496016e1e66e97c0c793b78245c44914f86b783558a45f5d36276da6c12bb7db38fdb570a9cc956b73a159d7e87aeadab63b357d134d5d4fb7a6867b6d04d6c2f76d2d5054855cf89f5fbcf8ec634d7ff0ec9986145509ef94f9f714a4cd10d2678631d395efabcada5fc04f356d6a6953cb9e3e34a6178fa60f1c780dbe9afe517eb4b80d936091ec2218a1af5dfb956d2aaab24defe24d106d82953287efd266c8716c63663a86237ff879b879b80b57cb19b235a4a199e66ada478aaabc861fc22fe4ef2d3cedab70b3dc7e152baa72b7f293eb6db4defb61316408464c55aefd75b87abbf71e72139fe34f92c1688248d635e4c99789bf59fad132751cff26d8a4835d7c28766b5509ee6e837510f9ab57e944f8d796efb846303516b639358dc572ea059e35f57de43bcec2763c04b790bef7da72960b5b8349d3cda9e97868ea3a863b359ceb45b0d091efbe366a27ad642d2980debd4b99f683923479331cd33cc3058f3933a6017262c0cd078fb651307b71b7f4936006ff962fae1e7f207f1526b772b4dbe03785cb7906a31c63355402830a17410ea1ef46fe3af86a1bbd3978e532f193e089bff16f82085cb2110a7d23293777fe665af64b6c45dd5c88098dc4bdef01a332a2b085016ec027d0f921499a6cc4414798f957bfbb9ba96673c395f6ddcdbf6bf026e32628eaaa7cc442bf8e7128ea75642230d0710440dd503de461516fe9c8d43351af57443d3d005e3db9941f46e1f226dcdcd407415b8e133f4ac4c6402c76269234f93771dca9f73892dac187c105f79fcca9b9e4d20dd3694eed98ad803db5bbf2a39b2079122c43ff3248521b09aea2f0e62688e42948e72809373772b225195fc090d9d935f653933a71c8ec9a0c0c9cb8df28031c0663c1a9d1bda52b19d38a46e3997d96052dfdf7c8e083fba6b954599861efef3ff213a4fdb0244dfe571cd278040f5ed838bd38345e84b0694fb50d6aca664d3eb734f3d5b9eedf80e9aecc2454d30b129dbbd4e1827e1f597a780e58ce81967d3bd47e8944c965836d9a60a0a2040bc1db8f48d2642b0e6fe4a6e157c4dc6649977996c149979da6ceb34a7b8dbc0243ec54a22af4513f706b5667f55a3cd35575de8dfdb8f8c5d3aac39cf2d40397a516b55b66b3043915b1d332c7c5034afba83309403855fc51499a58f7023c06b2b523e031555bd7b3b2806b5af2d3cfae3e797421237d1f3c34bd26ff59b009221f3279f9d1767db70a926059c8ae3a507eb2b98efc388976d81f8fb9f73e5c8875b549b09aa454d97069869d3b6369c6ab73dcbf9e62534efdb7eee8efc6143d076896e72e657d2b8a127cfc9858ddd2b7f91f8f9e787ef90d7ae7f66bfb00db6a7e3bc443ba4380eca0bf4217226508f58d92ba6b2676d880b9e6406220d5411609249e67b89982b58c7218a953b01f87f162fb6510bd6deeb21a7d15984976902976c3d03c703d51b902d869439b17eb961f97a4c917e2744b433da876b24d549eeabac9bef8d25fc532e45bf2e5dbcda2be9fe00cd14f8079ef17b9ef4b4f1d07066130e7b904081c0e06a3af4a131db23070f4576a030301134cb0181415650bc3423641cf4f48d2e45fc4a1e7b826a91fc936e39b69d7212db1b31aead784cb9aa43c0f43e6ef872ae5b0eb6e38ba253872f30894ec8d933478be5ac477cf82086c0bfc1fa268e766ab33da662be437e42ddcf27d2ca57e529226ff3d1a296568b96eb6cc3a29c5c5424eefabdd2f6b61094575a1851e8870c0e127643a878ff125d37531bd1c8ab06863a999d54d00014ac97dab0edbad9e77188a4c0d894d2231db7e4a92269138b611819a8946d3185bfc65a22b84c8c365df2949d912d5be5a6b4d650118e77e33d5e3f4ab73daba94262360f19169215620c64e4823dfd54125c5443fb964997e64324bee590c2376c86eecca6ea03b57097b7e5a92267f3d1c7b2c634cdadf562d0761f6d8a6e399596bad5a11ddab7d961a69c7c598c15b8be5977271f578ef3fe972c938deb6792005e4fbf1e58f4dddb934ba94455d6f828c7856b64a86da722f7bdb28a0d56f05000f288d776524d1 (caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 325700000800324300028e18785eec9dd992dc4857c70514fbbeef20c43233a02a94da55579fede98f9809dbe370b799083e7fe190abd4ddc2b5f448aa9931137e00b8011e051e00ee20d86e20e07988a34c2da54aa952aa544ae51edfd8eeaaae9232cff99dff392795a97f2a49d20f48d2e4dfbf51be9384eb204efcf59d3257744d37a69a37d5ac2b64ce0d776eba33647a7fa1a8ca77
Please advise on how to solve the above error ? Thank you.