In 2016 our cluster was setup using Node Client so that some nodes can function as pure coordinating nodes with additional custom REST API implementations. This was done to optimize on the extra network call for the custom REST API. This setup works fine but restricts us to use only the OSS distribution of elasticsearch which means that X-Pack features are not available.
Currently we are looking to finally migrate to the official REST Client. For this we have already migrated away from the not supported Node Client to the deprecated Transport Client and still using the official docker OSS distribution from elastic.
As the next steps we have tried to use the official elasticsearch docker distribution with Transport Client settings but failed to keep the cluster operational. Our attempts so far with the above mentioned setup have been:
Rolling out Master nodes before Data nodes: This was unsuccessful because data nodes didn't survive because the new master cluster was not recognized or rejected. The logs only report that a master node was not found.
Rolling out Data nodes first with Master nodes: This was also unsuccessful because the new master nodes were rejected. Again the underlying reason is not clear.
Does anyone have any recommendations or previous experiences?