Hello team!
I want to make use of the hot, warm, cold, frozen tiers for my onprem elastic deployment. I can see from this link that data tier is possible with platinum licensing.
However I see 'searchable snapshots' can be availed only with enterprise license.
My question is how will searchable snapshots help me if I can already do data tiering (cold and frozen) with platinum license? Please help!
Without the use of searchable snapshots, which allows very high node densities, there is basically no difference between the cold and frozen tier.
I want to understand the difference between 'searchable snapshots' and 'frozen tier'
Searchable snapshots is what makes a frozen tier frozen and allows it to hold higher densities of data than the cold tier. Without that there is no difference between a cold and frozen tier unless there is a difference in hardware resources configuration/resources. With the Platinum license you can define both cold and frozen tiers but they will basically behave the same way so there is in my opinion no need for a frozen tier without searchable snapshots.
1/ From this documentation, I see references of searchable snapshots in both cold and frozen tiers - Data tiers | Elasticsearch Guide [8.12] | Elastic
could you please let me know what 'searchable snapshots' actually mean
2/ with platinum license, data tiering is possible - meaning i can have hot, cold and frozen tier. Could you let me know what you mean by 'they basically behave the same way'. How does it behave with enterprise license and how does it behave with platinum license?
Tiering is available with the free Basic license and you can create hot, warm, cold and frozen (?) zones. This allows you to change index settings, e.g. compression and replica count, as well as have nodes with potentially different configuration serve the different tiers. This allows you to increase density or use chaper hardware for tiers holding older data in order to reduce cost. All these nodes still need to store the indices on local disk, and as each index has some overhead this restricts how much data a node can handle.
Searchable snapshots allow nodes to work with data stored in a snapshot repository, e.g. S3, and only store some data in a cache on local disk. This feature requires the Enterprise license but allows this type of node to handle a lot more data than a node storing data on local disks can handle.
Thanks for this!
Does this mean without searchable snapshots, local copies of frozen tiers need to be available in hot tier for it to be searchable? (what does local disk reference mean here?)
Can frozen tier (in platinum subscription) be searchable? If its searchable, what advantage does one get with searchable snapshots in enterprise license? (besides the need to store local copy of data), is there a difference in speed directly querying frozen tier and having searchable snapshots?
Data can be searched on all tiers, but as the amount of data held on the nodes and hardware resources typically vary between tiers, querying older data often comes with longer latencies.
Data on all tiers can be searched, but I am not sure how search speed varies, as this will depend on data volumes, queries and hardware resources.
The main benefit of searchable snapshots is that each node can hold and query a lot more data than nodes without this due to the lower amount of overhead.
Frozen tier is an Enterprise feature, there is no frozen tier with a platinum or basic license.
With a basic or platinum license you can have data tiering using hot, warm and cold nodes and for each tier you will need nodes specific configured for this using the correct roles, data_hot, data_warm and data_cold.
Please check this - frozen tier(data tiers) is supported in all tiers - Data tiers | Elasticsearch Guide [8.12] | Elastic
Not sure where you got this information, but as mentioned frozen tier is a enterprise feature, it does not exist for platinum or basic clusters.
The frozen tier requires the searchable snapshots feature, which is a paid feature only available at the Enterprise level.
Per the same documentation you linked you have this:
The frozen tier requires a snapshot repository. The frozen tier uses partially mounted indices to store and load data from a snapshot repository.
So, without searchable snapshots you cannot have the frozen tier as defined per Elastic.
On thing that can lead to confusion is that the searchable snapshots feature can also be used in other data tiers, like the cold tier.
This is also explained in the same linked documentation.
The difference is on the mount type of the searchable snapshots.
In resume, frozen tier requires searchable snapshots, but searchable snapshots can also be used in other data tiers, like hot and cold.
The link you provided says that the frozen tier relies on searchable snapshots, which is a feature only avalable with the Enterprise license. The definition used to be different and I was not aware of this change as I have not used searchable snapshots.
you can see how I have selected platinum subscription but still able to deploy hot, warm, cold and frozen tier. Searchable snapshot is an enterprise feature. Where does it say frozen tier is an enterprise feature?
" The frozen tier requires a snapshot repository. The frozen tier uses partially mounted indices to store and load data from a snapshot repository ."
Snapshot repository is different from searchable snapshots! snapshot repository is just to take backups of your data whereas searchable snapshots is different!
On the subscription page for the cloud offering.
I do not use Elastic Cloud, I have no idea why it shows a frozen storage with the Platinum license, since this needs an enterprise feature, you will need to wait what elastic answers.
Also, for the on-premises offering you have the information about searchable snapshots.
Searchable snapshot is a feature that allows you to query your data directly from snapshots, that's why the documentation says that the frozen tier requires a snapshot repository, because you need to have snapshots to be able to search on them.
The Frozen Tier is a data tier in the ILM process that works by using the searchable snapshots feature to search the data available in snapshots, those snapshots are stored on some snapshot repository, normally on a cloud service, like S3 for example.
So to be able to use the Frozen Tier you need to be able to use the feature of searchable snapshots, this feature is only available for a license at the Enterprise level.
What I would suggest is that you contact elastic directly and there's people like me that'll work through details of whether an enterprise license makes sense with your volume of data and a retention... It may or may not.
Trying to figure this all out on your own through the documentation and discuss can be a little bit challenging.
There's a common confusion and it's understandable that cold and frozen and searchable snapshots are exactly one in the same... They are not
Cold and frozen are node roles and content tiers
What makes them valuable and cost effective is searchable snapshots which is and enabling technology at the enterprise licensing.
You can set up and use a cold node without enterprise licensing and it will work fine. But you will not benefit from the underlying value of searchable snapshot.
The true value is the frozen tier
I'll just put it that simply this way
A normal data node is size at 10 to 12 terabytes of either SSD or HDD.
With a frozen node in searchable, snapshots 100 TB can live on S3 and be accessed from a single node adding much much cheaper cost.
That is the value of searchable snapshots.
So quick question
How many terabytes a day do you think you're ingesting and how long do you want to keep them?
The lowest level of how searchable snapshots are created and stored is managed automatically in elastic cloud and if you were to do on prem you would need to have your own S3 storage and set up some proper ILM polices.
And of course, if you were going to do that all on your own, you'll be working with elastic sales and they would probably suggest a couple days of professional services.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.
