TLS handshake failure between Logstash-LB and log drops in Logstash server

Elastic Stack Logstash
1

My architecture is Sources –> Load balancer –> Logstash server (8 CPU, 16 GB) –> Azure Sentinel.

For the store logs, we just did an extensive traffic analysis as well. In the connectivity between LB and Logstash cluster (2 servers) we identified that the TLS handshake is failing on port TCP 6514. Due to this event, connection draining is happening and less and less stores will be able to push syslog data. We could see that a TCP handshake is fine. But once the TLS Handshake start, we noticed a client-hello is being sent by the LB and acknowledged by the Logstash server. From this moment the communication stops.

Please find below the all the checks and findings from Logstash end:

  1. Verify that Logstash is listening on TCP/6514 with TLS enabled:

tcp 0 0 141.93.182.143:6514 0.0.0.0:* LISTEN 1052/java

  1. Logstash input configuration verify: There is no change till today in this input file.

  2. Verify Certificate validity: Checked fine

    4\. No permission/owner change on cert/key files.

  3. Intermediate CA verification: OK

    6\. Logstash pipeline is already optimized with modified number of workers to avoid queue overloading.    

7\. When Logstash and sender negotiate TLS repeatedly (especially with TLSv1.1, TLSv1.2, TLSv1.3 all allowed), one side can abort high lead or delay.  Hence, tested with TLS versions         TLSv1.1, TLSv1.2, TLSv1.3, but the issue persists in all cases.

  4. File descriptor changed to 65535

  5. Nothing in Logstash plain logs. SSL Debug log was enabled for last one day, but no single error/exception is observed till now related to TLS negotiations. No connection reset error observed also.

  6. Tested TLS handshake locally, run directly in Logstash server, but it got hang after connected:

    openssl s_client -connect 141.93.182.143:6514 -tls1_2
    CONNECTED(00000003)

  7. As per tcp dump analysis, it takes 107 seconds to send the sever-hello message, so this makes sense we don't see the server-hello on client side. but even though this flow has 146 seconds before it sends the server-hello to the client, but the clients allows it and a session seems to be established. but 100+ seconds for a server-hello = not done.

Please help to resolve this issue.