I have a long lived index on ElasticSearch for eg. events which is not partitioned by time. I can store about 3 months data in hot storage. After that I want to move it to warm. If I migrate the entire index to warm, do I have to now create a brand new index events_2 in hot and refer to the warm index as events. Can I setup a process once 3 months are over everyday one day of data be moved to warm? Can I create an index low_priority_index in UltraWarm storage and write to that while keeping important events in the events index?
For specific questions about ultra warm storage you will likely need to ask AWS support. On a more general level cold storage is especially useful for complete indices that are no longer updated or written to, e.g. time series data in time-based indices, where this types of cheaper low-performance can be used for rarely queries data. Based on your description it sounds like any kind of bold storage would be a bad fit unless you adopt time-based indices.
In my case an event once written can't be mutated. So if it is a read only store it works well for me. My question was more related to how do I manage my queries. For eg. if I have events index that I then move to ultrawarm every 3 months. The best option would be to be able to just move data that is more than 3 months old to ultrawarm. However, as per the docs I have to move an entire index and it doesn't just selectively move data older than 3 months to ultrawarm. So now my queries that used to query the events index will have to query a new index also events_1 (in ultrawarm). How can I manage that without changing my query code? Is using multi-target query the right way Multi-target syntax | Elasticsearch Reference [7.11] | Elastic ?
We don't support the aws fork sorry. You'd really need to ask them how this works with the APIs.
We do have [ILM]ILM: Manage the index lifecycle | Elasticsearch Reference [7.11] | Elastic), which manages all of that for you and takes away the complexity. You can get it via our Elasticsearch Service that's available on the AWS marketplace.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.