# Data fs grows forever after full snapshot is taken
**URL:** https://discuss.elastic.co/t/data-fs-grows-forever-after-full-snapshot-is-taken/353139
**Category:** Elasticsearch
**Tags:** snapshot-and-restore
**Created:** [February 13, 2024, 9:08am UTC](https://discuss.elastic.co/t/data-fs-grows-forever-after-full-snapshot-is-taken/353139 "2024-02-13T09:08:26Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@TinaMartiello](https://discuss.elastic.co/u/TinaMartiello)
#### Post date: [February 13, 2024, 9:08am UTC](https://discuss.elastic.co/t/data-fs-grows-forever-after-full-snapshot-is-taken/353139/1 "2024-02-13T09:08:26Z")
Hi,
we have 3 elasticsearch nodes, elasticsearch-8.11.1-1.x86\_64 and CentOS Linux release 7.3.1611 (Core) o.s. are installed on premis.
We have very busy read/write systems.
We are taking a full snapshot once a day, overwriting the existing one with the following cron command:
08 13 \* \* \* curl -XDELETE localhost:9200/\_snapshot/backup/backupfull; curl -XPUT localhost:9200/\_snapshot/backup/backupfull
The snapshot creation takes 10 minutes without any error.
The snapshot is taken on a separate fs, not the same fs where data is located.
Randomly, not all the time, after the snapshot is taken, the data fs starts growing.
The size of the index segments is still the same, is not growing, but the data filesystem is growing forever.
If we stop and start elasticsearch the space drop down to the usual occupation. If we close/open the indexes the space drop down as well.
We have decided to comment the snapshot crontab command and we are not experiencing this strange behaviour any more.
Can you help us to find out what is going on?
Thank you,
Tina
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [February 13, 2024, 9:16am UTC](https://discuss.elastic.co/t/data-fs-grows-forever-after-full-snapshot-is-taken/353139/2 "2024-02-13T09:16:41Z")
Hi Tina & welcome!
Sounds strange indeed. Can you capture the full contents of the data fs (e.g. run `ls -lR path/to/my/data`) just after taking the snapshot, and then again say 20min later when it's been growing for a while. Also `GET _segments` at the same times.
> [@TinaMartiello](#):
>
> the following cron command:
>
> 08 13 \* \* \* curl -XDELETE localhost:9200/\_snapshot/backup/backupfull; curl -XPUT localhost:9200/\_snapshot/backup/backupfull
This seems like a bad plan btw, don't delete your backup before creating the next one. For starters, it means there's a period of time where you have no backup at all, but also if you take today's snapshot _before_ deleting yesterday's snapshot then Elasticsearch will notice that most of the data hasn't changed which should make the process much quicker.
Also I'd recommend just using [SLM](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-take-snapshot.html#automate-snapshots-slm) rather than your own cron job. SLM is much more robust, e.g. it'll handle failures properly and won't delete your last-good snapshot.
---
### Author:  [@TinaMartiello](https://discuss.elastic.co/u/TinaMartiello)
#### Post date: [February 13, 2024, 9:32am UTC](https://discuss.elastic.co/t/data-fs-grows-forever-after-full-snapshot-is-taken/353139/3 "2024-02-13T09:32:19Z")
> [@DavidTurner](#):
>
> This seems like a bad plan btw, don't delete your backup before creating the next one. For starters, it means there's a period of time where you have no backup at all, but also if you take today's snapshot _before_ deleting yesterday's snapshot then Elasticsearch will notice that most of the data hasn't changed which should make the process much quicker.
the backup is saved on a separate device (rubrik backup) before the deletion, so we keep it. If we perform the snapshot before the deletion it failed because snapshot with the same name already exists.
I collect some stats and I will provide you.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [February 13, 2024, 9:51am UTC](https://discuss.elastic.co/t/data-fs-grows-forever-after-full-snapshot-is-taken/353139/4 "2024-02-13T09:51:50Z")
> [@TinaMartiello](#):
>
> the backup is saved on a separate device (rubrik backup) before the deletion, so we keep it
Rubrik claims to have some very clever deduplication functionality, but I wonder if it really works as well as the deduplication built into ES itself. In any case it's still a lot more work (including IO, blowing your page cache, and network traffic) for ES to take a full backup of everything every day.
> [@TinaMartiello](#):
>
> it failed because snapshot with the same name already exists.
If you call the snapshot `` then ES will include today's date in its name.
---
### Author:  [@TinaMartiello](https://discuss.elastic.co/u/TinaMartiello)
#### Post date: [February 13, 2024, 1:10pm UTC](https://discuss.elastic.co/t/data-fs-grows-forever-after-full-snapshot-is-taken/353139/5 "2024-02-13T13:10:42Z")
That is really a good point, we decided to follow your suggestions and use slm.
We scheduled a backup through kibana.
We keep an eye on it, hopefully we won't have any side effect.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 12, 2024, 1:11pm UTC](https://discuss.elastic.co/t/data-fs-grows-forever-after-full-snapshot-is-taken/353139/6 "2024-03-12T13:11:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.