Elastic Cloud on Azure

We will be rolling out our first production elastic cluster. At this stage we are looking at pros and cons of on prem vs cloud, however even when to comes to cloud we need to decide between provisioning on cloud.elastic.co into Azure or provisioning the elasticsearch solution from the Azure marketplace. Azure is our prefered cloud provider. Provisioning directly in Azure lets me select a closer data centre wheres provisioning from cloud.elastic into Azure only allows 2 different Azure regions. Also, if we provision directly in Azure we can use our existing tenant (and therefore the VPN connectivity into that tenant).

So, in theory, using the elasticsearch solution in the Azure marketplace seems preferable, but am I missing something?

Elasticsearch Service on Azure is hosted Elasticsearch and Kibana as part of Elastic Cloud. It's currently a beta release and available in two regions to begin, but expect more regions in the future.

Elasticsearch on the Azure Marketplace is an Azure Resource Manager (ARM) solution template offering that takes care of deploying and configuring Azure resources in your subscription, to setup an Elasticsearch cluster and optional Kibana and Logstash instances. The ARM template can also be deployed from the public GitHub repository.

There are many differences between the two, and it might be worthwhile getting in touch with us to have a conversation about them in more detail. Here are the biggest ones that come to mind, though this is a non-exhaustive list :slightly_smiling_face:

Elasticsearch cluster topologies and Availability

Elasticsearch Service is a hosted solution that deploys a cluster with Kibana, APM and Machine Learning in minutes. You can choose from a few different cluster topologies like High Memory, Hot/Warm, etc. to suit your use case. The cluster can be deployed for High Availability by deploying nodes across Availability Zones.

Elasticsearch ARM template can deploy clusters with dedicated master, data and coordinating nodes, and you can choose the VM SKUs for each. The configuration of each node role is homogeneous however i.e. all data nodes are configured with the same SKU, JVM heap, etc. so it is not possible to configure a Hot/Warm architecture out of the box. The ARM template deploys each set of node roles into its own Availability Set for High Availability. The template does not use Availability Zones since they are not available in all regions into which the template can be deployed.

Elastic Stack features, snapshots, scaling up/down, and upgrades

Elasticsearch Service comes bundled with The Elastic Stack features like Core security (Basic Authentication, TLS, RBAC), APM, Monitoring, plus others, which are configured by default. The Pricing tier chosen determines what Elastic Stack features are available to use. For example, being able to configure a SAML security realm for Single Sign-On with Azure Active Directory, or using the SQL ODBC client to connect Elasticsearch with other BI tools like Tableau, PowerBI, Excel, etc. requires the Platinum tier of Elasticsearch Service.

Cluster data is automatically snapshotted every 30 minutes, with varying snapshot retention policies.

Scaling up or down or Upgrading Elasticsearch Service is a single button click process, and the service takes care of all of the operations needed to perform the in-place operation in the background, without downtime.

Elasticsearch ARM template is a Bring-Your-Own-License (BYOL) solution. It deploys the default distribution that includes The Elastic Stack features, and the license level applied determines which features are available to use. The Azure Marketplace deployment always deploys with a 30-day trial license (Platinum features), and any license level from Basic to Platinum can be applied either before or after the trial expires.

Cluster data is not snapshotted by default. A template deployment from the command line using Azure CLI/PowerShell can be configured with repository-azure plugin to snapshot data to an Azure storage account, and once configured, snapshots are not automatically taken but would need to be manually triggered (or automated by you).

Upgrading a deployment from Elasticsearch ARM template is the same process as upgrading Elastricsearch on your own bare-metal or cloud provisioned infrastructure, requiring you to SSH into VMs, download the new version, and go through the upgrade process steps.

Scaling up a cluster requires provisioning new resources and including them in the cluster. Since the ARM template deploys in incremental mode by default, it is possible to scale up the number of data and coordinating nodes by deploying the exact same template and parameters (except for data or coordinating node count) into an existing resource group where a previous deployment has been performed. This is something that should be done only when dedicated master nodes are used. Scaling down a cluster is not supported.

Management, support and cost

Elasticsearch Service is monitored 24/7, and Elastic engineers are notified if network or hardware looks to be playing up. Depending on Pricing tier chosen, Elasticsearch Service can come with dedicated support with SLAs, from the folks that know and understand the products well. As a hosted service, you can focus on using the products to solve business problems, and let Elastic manage the infrastructure. The cost of Elasticsearch service can be reasonably calculated with the pricing calculator.

With Elasticsearch ARM template, you are responsible for managing all of the Azure resources deployed, which reside in one of your subscriptions. Depending on Subscription tier, a license/subscription can come with dedicated support with SLAs, from the folks that know and understand the products well. Since The Elastic Stack is deployed in your subscription, diagnostics and other metadata would be provided by you to Elastic engineers in a support scenario.

The cost of deploying the Elasticsearch ARM template will vary depending on the cluster topology deployed and the VM SKUs chosen for each of the VMs within the cluster. It will also vary depending on which Azure region you decide to deploy to. Because there are a number of variables involved, a cost needs to be calculated on a case-by-case basis. At a minimum though, you will pay for all Azure resources deployed by the template. The Azure pricing calculator can help in providing a reasonable estimate. In addition to the cost of Azure resources, Elastic Stack Gold and Platinum features require a license to be purchased, the cost of which is also dependent on a number of factors; it's best to get in touch with us to discuss.


Thank you. That helps a lot

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.