Elastic Cloud : Option to deploy into a single zone / alternative cloud providers

Elastic Stack Elasticsearch
1

We would like to ideally be able to avoid "inter-node" fees by ...

  • deploying all our nodes into a single zone (for the existing GCP/AWS/Azure workload), saving both elastic (and us) the inter networking fees.
  • deploying all our nodes into an alternative provider like Digital Ocean / Linode, who does not charge inter region data transfer fees.

Our current workload on elastic cloud, is high volume raw logs ingest (stored in hot for only 3 days), with very low search volume (which we can tolerate slow search). In addition, because filebeat knows how to automatically retry, we are able to tolerate temporary downtime.

In worse case scenario, of a zonal outage, we are comfortable with it, as there is already an hourly snapshot in the system.

Currently, when we did the typical 3 node HA cluster setup, we found that the "inter transfer" fees were significantly higher than our node cost.

Because we can handle the lack of HA, we ended up reconfiguring to a larger single node, single zone setup. Which eliminates the "inter transfer" fees.

However while doable, this makes scaling up/down a very troublesome process. Ideally, we would like to able to spin up 3 smaller nodes, in a single zone. In theory it's also less stable than 3 smaller nodes. (open to hardware failure, single instance crashes)

Alternatively, from what I understand, since Elastic Cloud is built on ECE - which requires S3 + Kubernetes. Cloud providers like Digital Ocean / Linode, "should work" as well.

1 Like
2

Welcome to our community! :smiley:

Thanks for posting this feedback. I don't have answers to your questions, but I have passed this onto the Cloud Product team.

3

Hi @devops_uilicious Welcome to the community.

Thanks for the feedback.

Yes agree there are use cases for more smaller nodes in a single zone...

Curious what size type nodes when you did 2 or 3 zones?

Curious about what is your daily ingest volume if you could
Share?

No worries if you can not.

4

@stephenb sorry for delayed response, didn't configure the relevant notifications.

During our initial experiments, where we routed "some of our logs".
We were using "45 GB storage, 1 GB RAM, Up to 2.5 vCPU " hot nodes in europe-west1 GCP.

As for daily ingest volume, it's hard to predict specifically due to compression. But we can estimate backwards from the bill.

So for about ~13GB of compressed logs / day, in 1 month it would be ...

  • 398.5 GB : GCP Data Transfer In (per GB) : $0
  • 6812.6 GB : GCP Data Transfer Inter-Node (per GB) : $76.63
  • 3 nodes standard price : $60.48
    (3 nodes is cheaper then 2 nodes, paradoxically)

I have no idea whats the compression ratio is, so it could be 26GB of logs a day, or higher. You might have some idea of whats the typical log compression ratio from log files be.

We have since changed to a single node to work around the issue, and scaled up to a single "360 GB storage, 8 GB RAM, Up to 4 vCPU" node

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.