# Elastic Cloud Log Archiving Recommendations

**URL:** <https://discuss.elastic.co/t/elastic-cloud-log-archiving-recommendations/264532>\
**Category:** Elasticsearch\
**Created:** [February 17, 2021, 8:25am UTC](https://discuss.elastic.co/t/elastic-cloud-log-archiving-recommendations/264532 "2021-02-17T08:25:34Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![dandrejvv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dandrejvv/32/84020_2.png) [@dandrejvv](https://discuss.elastic.co/u/dandrejvv)\
**Post date:** [February 17, 2021, 8:25am UTC](https://discuss.elastic.co/t/elastic-cloud-log-archiving-recommendations/264532/1 "2021-02-17T08:25:34Z")

</div>

We recently moved to the Elastic Stack on the Elastic Cloud but we have a very minimal setup and would like to ask for guidance and recommendations as to how to go about archiving logs without doing anything unnecessarily expensive.

So we have 1 node that does all the Elastic Search processing, it has 2 GB of RAM and 60 GB of space. We're already using 40 GB and would like to know how we can go about archiving the older logs instead of just deleting them. What would be the best way to go about this?

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [February 17, 2021, 9:02am UTC](https://discuss.elastic.co/t/elastic-cloud-log-archiving-recommendations/264532/2 "2021-02-17T09:02:22Z")

</div>

You have 2 options (as per my point of view) :

- Enable best\_compression on older indices, this will help to reduce used space
- Or do snapshot into an S3 bucket and delete old indices

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 18, 2021, 2:55am UTC](https://discuss.elastic.co/t/elastic-cloud-log-archiving-recommendations/264532/3 "2021-02-18T02:55:08Z")

</div>

Your other option is to setup a hot/warm/cold cluster and then migrate data onto cold nodes.

---

<div class="post-metadata">

**Author:** ![dandrejvv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dandrejvv/32/84020_2.png) [@dandrejvv](https://discuss.elastic.co/u/dandrejvv)\
**Post date:** [February 19, 2021, 11:33am UTC](https://discuss.elastic.co/t/elastic-cloud-log-archiving-recommendations/264532/4 "2021-02-19T11:33:08Z")

</div>

Thanks for the reply, where can I find an article that explains how to setup this "best\_compression"?

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [February 19, 2021, 11:35am UTC](https://discuss.elastic.co/t/elastic-cloud-log-archiving-recommendations/264532/5 "2021-02-19T11:35:29Z")

</div>

```auto
PUT /index_name/_setting
{
    "index": {
      "codec": "best_compression"
    }
}

```

---

<div class="post-metadata">

**Author:** ![dandrejvv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dandrejvv/32/84020_2.png) [@dandrejvv](https://discuss.elastic.co/u/dandrejvv)\
**Post date:** [February 26, 2021, 11:01am UTC](https://discuss.elastic.co/t/elastic-cloud-log-archiving-recommendations/264532/6 "2021-02-26T11:01:09Z")

</div>

I noticed that I cannot execute this on an index that is already open. How is this meant to be used?

---

<div class="post-metadata">

**Author:** ![dandrejvv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dandrejvv/32/84020_2.png) [@dandrejvv](https://discuss.elastic.co/u/dandrejvv)\
**Post date:** [February 26, 2021, 11:03am UTC](https://discuss.elastic.co/t/elastic-cloud-log-archiving-recommendations/264532/7 "2021-02-26T11:03:32Z")

</div>

Thanks for the reply. I noticed now the options of adding a warm / cold cluster in the cloud. I noticed that the costing on a cold cluster is quite large compared to the hot node. If I don't query old data enough, will that mean I pay less overall?

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [February 26, 2021, 11:14am UTC](https://discuss.elastic.co/t/elastic-cloud-log-archiving-recommendations/264532/8 "2021-02-26T11:14:12Z")

</div>

This can be donne only when creating new index  
Use it in an ILM for example  
Or reindex old data into a new index with this seting

---

<div class="post-metadata">

**Author:** ![dandrejvv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dandrejvv/32/84020_2.png) [@dandrejvv](https://discuss.elastic.co/u/dandrejvv)\
**Post date:** [February 27, 2021, 8:15pm UTC](https://discuss.elastic.co/t/elastic-cloud-log-archiving-recommendations/264532/9 "2021-02-27T20:15:55Z")

</div>

I don't suppose I can run a task on ILM or something on Elastic Cloud that can do the re-indexing for me while enabling the best\_compression option? All I can think of is actually running a long-running script to do that for me.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 27, 2021, 8:16pm UTC](https://discuss.elastic.co/t/elastic-cloud-log-archiving-recommendations/264532/10 "2021-03-27T20:16:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.