Logstash not ready, when output goes down and then comes up during certificate renewal

I was running a test scenario where I was using TTL period 15 mins. After first 15 mins, certificate got renewed, then I scaled down elasticsearch to zero and waited for another 15 mins, it showed that elasticsearch HostUnreachable error, at 15 mins which is fine, but after scaling it up again, it continued to show the same, and went to Readiness probe failure and logstash didn't come up after that. Expecting it to come up after I scale up elasticsearch.

{"version":"1.2.0","timestamp":"2023-10-18T10:38:48.546Z","severity":"info","service_id":"eric-log-transformer","message":"Readiness probe failed: curl: (7) Failed to connect to localhost port 8080 after 0 ms: Connection refused\n exit status 7","metadata":{"pod_name":"eric-log-transformer-c4f457f88-cpzz9","container_name":"logtransformer","namespace":"zyadros"}}
{"version": "1.1.0", "timestamp": "2023-10-18T10:38:48.786Z", "severity": "warning", "service_id": "eric-log-transformer", "metadata" : {"namespace": "zyadros", "pod_name": "eric-log-transformer-c4f457f88-cpzz9", "node_name": "node-10-63-142-139", "container_name": "logtransformer"}, "message": "Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>'https://eric-data-search-engine-tls:9200/', :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>'OpenSearch Unreachable: [https://eric-data-search-engine-tls:9200/][Manticore::ClientProtocolException] Received fatal alert: certificate_unknown'}"}

Any suggestions will be helpful. Thanks in advance.

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

You have a certificate error, you need to troubleshoot this and fix it.

Where did you change the certificate? In elasticsearch only? You need to also change the certificates in Logstash.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.