Logstash TLS/SSL fails to connect to Elasticsearch with CN=instance does not match the certificate subject provided

[Bharath_venkat]

Hi

Trying to enable logstash TLS/SSL communication on kubernetes. Logstash fails to communicate with Elasticsearch . ried several ways to fix the CN instance issue . below are my methods

Method 1

Created cert bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --dns sample-elasticsearch-0
sample-elasticsearch-0 is the hostname of the elasticsearch

Blockquote
[2019-12-09T18:59:26,079][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://filebeat:xxxxxx@sample-elasticsearch-0.elasticsearch.svc.cluster.local:9200/", :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://filebeat:xxxxxx@sample-elasticsearch-0.elasticsearch.svc.cluster.local:9200/][Manticore::ResolutionFailure] sample-elasticsearch-0.elasticsearch.svc.cluster.local"}

when i do curl on the pod

sh-4.2# curl -k -u filebeat:123456 https://sample-elasticsearch-0.elasticsearch.svc.cluster.local:9200 curl: (6) Could not resolve host: sample-elasticsearch-0.elasticsearch.svc.cluster.local; Unknown error

Method 2

I have a service sample-elasticsearch-svc which is attached to sample-elasticsearch-0 pod . When i Curl from the pod i get response as below . After changing logstash config host to sample-elasticsearch-svc the logstash pods fails with CN instance issue because the elastic hostname is sample-elasticsearch-0 but here i am pointing to the k8 service

Blockquote sh-4.2# curl -k -u filebeat:123456 https://sample-elasticsearch-svc.elasticsearch.svc.cluster.local:9200
{
"name" : "sample-elasticsearch-0",
"cluster_name" : "sample-elasticsearch-cluster",
"cluster_uuid" : "UrTJQee6QoSD2TEPketiMw",
"version" : {
"number" : "7.4.1",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "fc0eeb6e2c25915d63d871d344e3d0b45ea0ea1e",
"build_date" : "2019-10-22T17:16:35.176724Z",
"build_snapshot" : false,
"lucene_version" : "8.2.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

Logs on Pod

Blockquote arch-svc.elasticsearch.svc.cluster.local:9200/]}}
[2019-12-09T19:09:41,088][ERROR][logstash.javapipeline ] Pipeline aborted due to error {:pipeline_id=>"beats", :exception=>#<Manticore::UnknownException: Host name 'sample-elasticsearch-svc.elasticsearch.svc.cluster.local' does not match the certificate subject provided by the peer (CN=instance)>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:37:in block in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:79:in call

config for logstash

Blockquote
data:
logstash.yml: |-
xpack.monitoring.elasticsearch.hosts: https://sample-elasticsearch-0.elasticsearch.svc.cluster.local:9200
dead_letter_queue.enable: true
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: filebeat
xpack.monitoring.elasticsearch.password: "123456"
xpack.monitoring.elasticsearch.ssl.verification_mode: none
xpack.monitoring.elasticsearch.ssl.certificate_authority: "/usr/share/logstash/config/elastic-stack-ca.pem"
output {
elasticsearch {
hosts => "sample-elasticsearch-0.elasticsearch.svc.cluster.local:9200"
manage_template => false
user => 'filebeat'
password => '123456'
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
ssl => true
cacert => "/usr/share/logstash/config/elastic-stack-ca.pem"
}
}

Elasticsearch config

cluster.name: "sample-elasticsearch-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1
# Update max_local_storage_nodes value based on number of nodes
node.max_local_storage_nodes: 1
xpack.security.enabled: true
xpack.monitoring.collection.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.license.self_generated.type: trial
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/elastic-certificates.p12
http.cors.enabled: true
http.cors.allow-origin: "*"
http.max_header_size: 16kb
cluster.initial_master_nodes:
 - sample-elasticsearch-0

I would really appreciate any help here