# S3 API Costs are extraordinary expensive for snapshots

**URL:** <https://discuss.elastic.co/t/s3-api-costs-are-extraordinary-expensive-for-snapshots/329071>\
**Category:** Elasticsearch\
**Tags:** slm-snapshot-lifecycle-management, snapshot-and-restore\
**Created:** [March 31, 2023, 5:25pm UTC](https://discuss.elastic.co/t/s3-api-costs-are-extraordinary-expensive-for-snapshots/329071 "2023-03-31T17:25:34Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Doc\_Kaos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doc_kaos/32/53671_2.png) [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)\
**Post date:** [March 31, 2023, 5:25pm UTC](https://discuss.elastic.co/t/s3-api-costs-are-extraordinary-expensive-for-snapshots/329071/1 "2023-03-31T17:25:34Z")

</div>

To store 5TB of data, we are paying about $1,200 in storage fees per month and $10,000 in API calls

Is there a way to fix this? During a snapshot we are seeing upwards of 120k s3 api calls/minute

SLM:

```auto
PUT _slm/policy/hourlies
{
  "name": "<hourly-snap-{now/d}>",
  "schedule": "0 0 * * * ?",
  "repository": "s3backup",
  "config": {
    "ignore_unavailable": true,
    "include_global_state": true,
    "feature_states": [],
    "partial": true
  },
  "retention": {
    "expire_after": "14d",
    "min_count": 1,
    "max_count": 24
  }
}

```

And the repo is:

```auto
  "s3backup": {
    "type": "s3",
    "uuid": "UUID",
    "settings": {
      "bucket": "bucketname",
      "endpoint": "s3.us-thing-1.amazonaws.com",
      "server_side_encryption": "true",
      "max_restore_bytes_per_sec": "500mb",
      "storage_class": "intelligent_tiering",
      "use_throttle_retries": "true",
      "readonly": "false",
      "base_path": "cluster_name",
      "region": "us-thing-1",
      "max_snapshot_bytes_per_sec": "500mb"
    }
  }
}

```

This cluster is running 8.6.2,  
Nodes: 24  
Indices: 5076  
Documents: 850M  
Disk Usage: 1.4TB  
Primary Shards: 15k  
Replica Shards: 15k

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [March 31, 2023, 6:01pm UTC](https://discuss.elastic.co/t/s3-api-costs-are-extraordinary-expensive-for-snapshots/329071/2 "2023-03-31T18:01:04Z")

</div>

> [@Doc\_Kaos](#):
>
> Disk Usage: 1.4TB  
> Primary Shards: 15k  
> Replica Shards: 15k

API calls during snapshots scale approximately with the number of shards, and that is a tiny amount of data for such a high shard count.

---

<div class="post-metadata">

**Author:** ![Doc\_Kaos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doc_kaos/32/53671_2.png) [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)\
**Post date:** [March 31, 2023, 6:21pm UTC](https://discuss.elastic.co/t/s3-api-costs-are-extraordinary-expensive-for-snapshots/329071/3 "2023-03-31T18:21:56Z")

</div>

I agree 😃

Unfortunately it's a legacy model ☹ If there is no other option, maybe I'll look at convincing people it's worth refactoring ...

Our search rate is quite high which maybe an issue for less shards.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/d/ad5f83859c68e5f8d64dda7966e7d79286e7d65b.png)

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [March 31, 2023, 7:04pm UTC](https://discuss.elastic.co/t/s3-api-costs-are-extraordinary-expensive-for-snapshots/329071/4 "2023-03-31T19:04:45Z")

</div>

When there's up to $10k/mo of savings up for grabs, hopefully it's an easy sell 🙂

The only other option I can think of is to replace S3 with something that doesn't charge per request, maybe a self-managed NFS or Minio. But then you've got to think about the admin hassle.

100k searches per sec across 24 nodes doesn't sound especially high to me, although it very much depends on the details of your workload. You might even get better performance with fewer shards. Make sure to use [Rally](https://esrally.readthedocs.io/en/stable/index.html) to benchmark your experiments.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 28, 2023, 7:05pm UTC](https://discuss.elastic.co/t/s3-api-costs-are-extraordinary-expensive-for-snapshots/329071/5 "2023-04-28T19:05:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.