We would like to ideally be able to avoid "inter-node" fees by ...
- deploying all our nodes into a single zone (for the existing GCP/AWS/Azure workload), saving both elastic (and us) the inter networking fees.
- deploying all our nodes into an alternative provider like Digital Ocean / Linode, who does not charge inter region data transfer fees.
Our current workload on elastic cloud, is high volume raw logs ingest (stored in hot for only 3 days), with very low search volume (which we can tolerate slow search). In addition, because filebeat knows how to automatically retry, we are able to tolerate temporary downtime.
In worse case scenario, of a zonal outage, we are comfortable with it, as there is already an hourly snapshot in the system.
Currently, when we did the typical 3 node HA cluster setup, we found that the "inter transfer" fees were significantly higher than our node cost.
Because we can handle the lack of HA, we ended up reconfiguring to a larger single node, single zone setup. Which eliminates the "inter transfer" fees.
However while doable, this makes scaling up/down a very troublesome process. Ideally, we would like to able to spin up 3 smaller nodes, in a single zone. In theory it's also less stable than 3 smaller nodes. (open to hardware failure, single instance crashes)
Alternatively, from what I understand, since Elastic Cloud is built on ECE - which requires S3 + Kubernetes. Cloud providers like Digital Ocean / Linode, "should work" as well.