Hi @DavidTurner
I would like to clarify a critical operational constraint that may not be obvious:
Cluster Size: Our production cluster has 1000+ nodes (testing is performed on an 18-node subset).
The Restart Problem:
Adding explicit truststore configuration requires:
-
Updating elasticsearch.yml on each node
-
Restarting the Elasticsearch service (config changes require restart)
-
Node leaves cluster during restart
-
Node rejoins cluster after restart
Why This Creates Downtime:
If we deploy truststore in a rolling fashion:
-
Nodes without truststore (implicit trust): 990 nodes online
-
Nodes with truststore (explicit trust): 10 nodes restarting/rejoining
When the 10 restarted nodes try to rejoin, they fail SSL handshake with the 990 nodes (as we've observed in testing). They remain isolated until ALL 1000+ nodes have truststore deployed.
Rolling deployment: Requires nodes with different trust configurations to coexist → SSL handshake fails → nodes can't communicate
Big deployment: Restart all 1000+ nodes simultaneously → guaranteed cluster-wide outage
This is why we're seeking confirmation: Is there ANY way to deploy truststore without requiring all nodes to restart simultaneously?
The original motivation was zero-downtime certificate rotation, but if truststore deployment itself requires downtime, it defeats the purpose.