Hi all. I know this has been discussed but I still can't figure it out. Getting the "unable to verify the first certificate" in the logs when starting Kibana.
Here are the logs, the part where the errors start showing, later it keeps looping with "Unable to revive connection".
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["info","savedobjects-service"],"pid":12207,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["error","elasticsearch","data"],"pid":12207,"message":"Request error, retrying\nHEAD https://es6.myhost.internal:9200/.apm-custom-link => unable to verify the first certificate"}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["error","elasticsearch","data"],"pid":12207,"message":"Request complete with error\nGET https://es6.myhost.internal:9200/_xpack => unable to verify the first certificate"}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["warning","plugins","licensing"],"pid":12207,"message":"License information could not be obtained from Elasticsearch due to Error: Error: unable to verify the first certificate error"}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["debug","plugins","licensing"],"pid":12207,"message":"Imported license information from Elasticsearch:type: undefined | status: undefined | expiry date: Invalid date"}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["info","plugins","searchprofiler"],"pid":12207,"message":"You cannot use searchprofiler because license information is not available at this time."}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["info","plugins","painlessLab"],"pid":12207,"message":"You cannot use painlessLab because license information is not available at this time."}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["info","plugins","snapshotRestore"],"pid":12207,"message":"You cannot use snapshot_restore because license information is not available at this time."}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["info","plugins","transform"],"pid":12207,"message":"You cannot use transform because license information is not available at this time."}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["info","plugins","indexManagement"],"pid":12207,"message":"You cannot use index_management because license information is not available at this time."}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["info","plugins","remoteClusters"],"pid":12207,"message":"You cannot use Remote Clusters because license information is not available at this time."}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["info","plugins","watcher"],"pid":12207,"message":"You cannot use watcher because license information is not available at this time."}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":12207,"message":"Monitoring status upload endpoint is not enabled in Elasticsearch:Monitoring stats collection is stopped"}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["error","elasticsearch","data"],"pid":12207,"message":"Request complete with error\nHEAD https://es6.myhost.internal:9200/.apm-agent-configuration => unable to verify the first certificate"}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["warning","elasticsearch","data"],"pid":12207,"message":"Unable to revive connection: https://es2.myhost.internal:9200/"}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["error","elasticsearch","admin"],"pid":12207,"message":"Request error, retrying\nGET https://es3.myhost.internal:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => unable toverify the first certificate"}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["warning","elasticsearch","data"],"pid":12207,"message":"Unable to revive connection: https://es8.myhost.internal:9200/"}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["error","elasticsearch","admin"],"pid":12207,"message":"Request error, retrying\nGET https://es7.myhost.internal:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => unable toverify the first certificate"}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["warning","elasticsearch","data"],"pid":12207,"message":"Unable to revive connection: https://es7.myhost.internal:9200/"}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["error","elasticsearch","admin"],"pid":12207,"message":"Request error, retrying\nGET https://es5.myhost.internal:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => unable toverify the first certificate"}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["warning","elasticsearch","data"],"pid":12207,"message":"Unable to revive connection: https://es4.myhost.internal:9200/"}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["error","elasticsearch","admin"],"pid":12207,"message":"Request complete with error\nGET https://es8.myhost.internal:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => unable to verify the first certificate"}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["error","savedobjects-service"],"pid":12207,"message":"Unable to retrieve version information from Elasticsearch nodes."}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["warning","elasticsearch","data"],"pid":12207,"message":"Unable to revive connection: https://es1.myhost.internal:9200/"}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["warning","elasticsearch","data"],"pid":12207,"message":"Unable to revive connection: https://es5.myhost.internal:9200/"}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["warning","elasticsearch","data"],"pid":12207,"message":"Unable to revive connection: https://es3.myhost.internal:9200/"}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["warning","elasticsearch","data"],"pid":12207,"message":"Unable to revive connection: https://es6.myhost.internal:9200/"}
{"type":"log","@timestamp":"2022-06-02T19:16:16Z","tags":["warning","elasticsearch","data"],"pid":12207,"message":"No living connections"}
Here is the Elasticsearch.yml
cluster.name: mycluster
node.name: es1
path.data: /opt/elastic/data
path.logs: /opt/elastic/logs
network.host: es1.myhost.internal
http.port: 9200
discovery.seed_hosts: ["es1.myhost.internal", "es2.myhost.internal", "es3.myhost.internal", "es4.myhost.internal", "es5.myhost.internal", "es6.myhost.internal", "es7.myhost.internal", "es8.myhost.internal"]
cluster.initial_master_nodes: ["es1", "es2", "es3"]
node.master: true
node.ingest: true
node.data: false
node.ml: false
node.transform: false
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.keystore.path: /opt/elastic/elasticsearch/config/certs/xeno-keystore-dev1.jks
xpack.security.transport.ssl.truststore.path: /opt/elastic/elasticsearch/config/certs/xeno-trust-dev1.jks
xpack.security.http.ssl.keystore.path: /opt/elastic/elasticsearch/config/certs/xeno-keystore-dev1.jks
xpack.security.http.ssl.truststore.path: /opt/elastic/elasticsearch/config/certs/xeno-trust-dev1.jks
And kibana.yml
server.port: 5601
server.host: es1.myhost.internal
elasticsearch.hosts: ["https://es1.myhost.internal:9200", "https://es2.myhost.internal:9200", "https://es3.myhost.internal:9200", "https://es4.myhost.internal:9200", "https://es5.myhost.internal:9200", "https://es6.myhost.internal:9200", "https://es7.myhost.internal:9200", "https://es8.myhost.internal:9200"]
server.ssl.enabled: true
elasticsearch.ssl.verificationMode: full
server.ssl.keystore.path: /opt/elastic/kibana/config/xeno-keystore-dev1.p12
elasticsearch.ssl.keystore.path: /opt/elastic/kibana/config/xeno-keystore-dev1.p12
elasticsearch.ssl.truststore.path: /opt/elastic/kibana/config/xeno-truststore-dev1.p12
elasticsearch.ssl.certificateAuthorities: ["/opt/elastic/kibana/config/elasticsearch-ca.pem"]
elasticsearch.username: "kibana"
elasticsearch.password: "mypassword"
logging.verbose: true
logging.dest: /opt/elastic/logs/kibana.log
xpack.security.encryptionKey: "mystringsof32chars"
No issues with certificates on Elasticsearch, I can curl with "elastic" user and password:
curl -ik -u elastic https://es1.myhost.internal:9200/_cat/health?v
Any suggestions? Thanks