How to use frozen storage the right way

[axel_r (Axel)]

Hello everyone,

I'd like some clarification and information on data tiers frozen in an Elastic Cloud environment.
I'm not sure if I understand how this works and I can't find any documentation that answers my questions.

From what I understand, and from what I've been able to exchange with Elastic architects, frozen storage enables snapshots to be stored and partially mounted in a cache.

The problem I'm having is that when I configure my ILM Policy, I'm asked to specify a "Snapshot repository".
And the only selection I have is about managed storage from EC, which are therefore based on a different storage than frozen and are therefore billed in a different way :

My ILM policy includes only hot and frozen data tiers

How can I configure an ILM Policy with a frozen data-tier to use the 6.25 TB and the 380 GB of cached storage advertised on the EC instance edit screen ?

Thanks for your help

[stephenb (Stephen Brown)]

Hi @axel_r Welcome to the community and thanks for Trying Elastic Cloud and Frozen which is pretty amazing.

So if you select this in your Edit Page

And this in the ILM

You are setup and ready to go!

@axel_r

The problem I'm having is that when I configure my ILM Policy, I'm asked to specify a "Snapshot repository".
And the only selection I have is about managed storage from EC, which are therefore based on a different storage than frozen and are therefore billed in a different way :

Let me see if I can clarify.

The Provided Snapshot repository is used to store both Normal Snapshots and Searchable Snapshots which are used with Frozen.

That snapshot storage is charged at the same rate no matter how it is used. It is blob storage and the charges can be found in the Data Transfer and Storage portion of the Usage / Billing Screens

When you select a Fozen node like above you are charged for that node ... not all the "potential storage" for the searchable snapshots.

That screen can be a bit misleading what you are paying for is

A node that has 4GB RAM, up to 2.5vCPU and if you look closely 380GB of SSD Disk which is used at the local cache for the Frozen / Searchable Snapshot. That is what you are paying for.

"Includes 380 GB of cached storage for 6.25 TB of storage."

When you Provision the Frozen Node you are NOT paying for 6.25TB of space, that is the Total Maximum mountable Searchable Snapshot that that Node can use The Difference is about 5.87 TB of the S3 / Bob storage for the Searchable Snapshot. You will only be charged for that at the rate of the Snapshot Storage AS you create the Searchable Snapshot.

In the end, this is extremely cost effective.

Hope that helps.

Here is where you will see your Data Transfer and Storage Charges

[axel_r (Axel)]

Hello Stephen,

Thanks for your feedback!

I'm having a bit of trouble understanding what I'm paying for and where this architecture saves me money. Sorry, but English is not my main language.

Let me summarize :

@stephenb

That snapshot storage is charged at the same rate no matter how it is used. It is blob storage and the charges can be found in the Data Transfer and Storage portion of the Usage / Billing Screens

The snapshots created by the ILM Policy are always stored in snapshot storage.
I pay for storage per GB-month and per batch of 1K requests

@stephenb

When you Provision the Frozen Node you are NOT paying for 6.25TB of space, that is the Total Maximum mountable Searchable Snapshot that that Node can use The Difference is about 5.87 TB of the S3 / Bob storage for the Searchable Snapshot.

I pay for a frozen node for 380 Gb of data
This node allows up to 5.87 TB of data to be viewed/consulted (?)

@stephenb

You will only be charged for that at the rate of the Snapshot Storage AS you create the Searchable Snapshot.

So I'm charged the price of the frozen instance in all cases + the price per GB-month only when I create Searchable Snapshots. But I also pay for the time the snapshot is stored in the Snapshot Storage ?
I no longer really understand the purpose of frozen storage, is it only used to mount searchable snapshots ? and I have to include the cost of Snapshot Storage in my calculations ?

Thanks for your help!

[stephenb (Stephen Brown)]

@axel_r

The snapshots created by the ILM Policy are always stored in snapshot storage.
I pay for storage per GB-month and per batch of 1K requests

Correct. There is a free allows of API Calls as well.

@axel_r

I pay for a frozen node for 380 Gb of data
This node allows up to 5.87 TB of data to be viewed/consulted (?)

Correct
380GB is High Speed SSD
Up To ~5.87 TB of Snapshot Storage as used (Very Inexpensive)

@axel_r

So I'm charged the price of the frozen instance in all cases + the price per GB-month only when I create Searchable Snapshots. But I also pay for the time the snapshot is stored in the Snapshot Storage ?

I no longer really understand the purpose of frozen storage, is it only used to mount searchable snapshots ? and I have to include the cost of Snapshot Storage in my calculations ?

1st you do not pay for the snapshot storage twice... just once when it is created (and for the time you keep it) There is no additional cost when it is Partially mounted to the Frozen node and becomes searchable.

2nd why this makes it cost-effective is that if you needed/wanted to store the 5.87TB on SSD (Hot) or HDD (Warm) that would be many times more expensive that 5.87TB of snapshot storage + the Frozen Node cost

We have customers that want to store 100TBs of logs... the Snapshot storage is only a small fraction of the cost if they tried to store that all on SSD or HDD.

Hope that makes sense.

You might want to take a look at this as well... its a bit older but concepts remain the same

Some Quick Math (if I did this right)
Say I wanted
6TB of Storage capacity on the Warm Tier is about $1400 / month (HDD)
6TB of Storage Capacity on Frozen = ~$153 for the 4GB Node (includes 380 GB SSD) + $193 for the Snapshot Storage = $346 / month

The difference between Hot and Frozen is even larger...