# Backup repository size is much bigger than indices size
**URL:** https://discuss.elastic.co/t/backup-repository-size-is-much-bigger-than-indices-size/47469
**Category:** Elasticsearch
**Created:** [April 15, 2016, 7:28am UTC](https://discuss.elastic.co/t/backup-repository-size-is-much-bigger-than-indices-size/47469 "2016-04-15T07:28:51Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@Faye](https://discuss.elastic.co/u/Faye)
#### Post date: [April 15, 2016, 7:28am UTC](https://discuss.elastic.co/t/backup-repository-size-is-much-bigger-than-indices-size/47469/1 "2016-04-15T07:28:51Z")
I have an Elasticsearch backup per hour, after about 370 times of backup (about 15 days) , my backup repository is more than 15G !!! But the total indices size is just about 500M !! Elasticsearch is incremental backup, but 15G VS 500M , the difference is so huge ! I wonder whether it is normal with so big different size between indices and backup repository ?
Is it caused by my frequent backup (hourly) ? I use the hourly backup in cluster 1 and hourly restore in cluster 2 to keep two ES clusters data same real time .
======
My Elasticsearch settings : 2 nodes , 12 shard/node , 2 indices , fs type of backup to store snapshots to NAS
in Elasticsearch data directory , the indices size :
# node1 indices size
[root@esnode1 indices]$ du -sh
266M .
# node2 indices size
[root@esnode2 indices]$ du -sh
238M .
# in backup repository , the size :
[root@esnode1 backup]$ du -lh
114M ./backup/indices/index1/0
112M ./backup/indices/index1/5
114M ./backup/indices/index1/11
114M ./backup/indices/index1/10
111M ./backup/indices/index1/8
116M ./backup/indices/index1/4
120M ./backup/indices/index1/9
118M ./backup/indices/index1/3
114M ./backup/indices/index1/2
115M ./backup/indices/index1/7
115M ./backup/indices/index1/1
112M ./backup/indices/index1/6
1.4G ./backup/indices/index1
747M ./backup/indices/index2/0
1.6G ./backup/indices/index2/5
887M ./backup/indices/index2/11
743M ./backup/indices/index2/10
2.1G ./backup/indices/index2/8
801M ./backup/indices/index2/4
1.3G ./backup/indices/index2/9
878M ./backup/indices/index2/3
951M ./backup/indices/index2/2
1.2G ./backup/indices/index2/7
953M ./backup/indices/index2/1
943M ./backup/indices/index2/6
13G ./backup/indices/index2
15G ./backup/indices
15G ./backup
1.1M ./backuplogs
15G .
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 15, 2016, 7:54am UTC](https://discuss.elastic.co/t/backup-repository-size-is-much-bigger-than-indices-size/47469/2 "2016-04-15T07:54:23Z")
Snapshot and restore works at the segment level and is incremental in that it will only snapshot a segment once even if it is used in multiple snapshots. This is described quite well [in this blog post](https://www.elastic.co/blog/found-elasticsearch-snapshot-and-restore). As segments merge, these new segments will also be backed up, and as there will be multiple segments that hold the same records, snapshotting is not incremental at the record level.
---
### Author:  [@Faye](https://discuss.elastic.co/u/Faye)
#### Post date: [April 15, 2016, 8:40am UTC](https://discuss.elastic.co/t/backup-repository-size-is-much-bigger-than-indices-size/47469/3 "2016-04-15T08:40:05Z")
yes, I searched and read the blog you posted from other's topic before posting this one , My case does not touch segment merge ....
I'm just curious the backup repository size is so huge (15G) compared with the indices data (500M) . almost 1 G increasement every day , it will eat up my NAS soon .
---
### Author:  [@Faye](https://discuss.elastic.co/u/Faye)
#### Post date: [April 15, 2016, 8:41am UTC](https://discuss.elastic.co/t/backup-repository-size-is-much-bigger-than-indices-size/47469/4 "2016-04-15T08:41:37Z")
Now that it is incremental backup , my backup size should be similar to indices size or at most double it since there are some snapshost record files
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 15, 2016, 9:04am UTC](https://discuss.elastic.co/t/backup-repository-size-is-much-bigger-than-indices-size/47469/5 "2016-04-15T09:04:14Z")
If you index into Elasticsearch, segments will automatically be merged in the background.
---
### Author:  [@Faye](https://discuss.elastic.co/u/Faye)
#### Post date: [April 15, 2016, 11:04am UTC](https://discuss.elastic.co/t/backup-repository-size-is-much-bigger-than-indices-size/47469/6 "2016-04-15T11:04:26Z")
ah , I did not realize the segments merge will automatically happen as long as there is indexing to ES
Now that , I tried by following the "Merge" part in posted blog , after merge , when backup again , the repository size increases more than double . Now sure how often merge will happen when index into ES , if one index to ES , one merge happens, then if I backup , absolutely the size of repository will be very huge . so It is expected that a large of backup repository size , right ?
I am using the backup and restore for the data sync between our production system and disaster recovery system , backup/restore happens hourly , that is to say , I have to delete the previous snapshot by API termly , right ?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 15, 2016, 12:13pm UTC](https://discuss.elastic.co/t/backup-repository-size-is-much-bigger-than-indices-size/47469/7 "2016-04-15T12:13:29Z")
Deleting old snapshots will remove segments that no snapshot longer refer to, and will reduce storage space. How many old snapshots do you need to keep? How far back in time do you need to be able to restore?
---
### Author:  [@Faye](https://discuss.elastic.co/u/Faye)
#### Post date: [April 18, 2016, 5:53am UTC](https://discuss.elastic.co/t/backup-repository-size-is-much-bigger-than-indices-size/47469/8 "2016-04-18T05:53:18Z")
I think two months are enough from business consideration . So I can delete the old snapshots every two months.
Just want to confirm with you again
1. It is a normal result that the indexes and backup repository have a big different size (500G VS 15G) in my case, right ?
2. Some of redundant data in backup snapshots are caused by segment merge of Lucene , right ?
Thanks !
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 18, 2016, 6:05am UTC](https://discuss.elastic.co/t/backup-repository-size-is-much-bigger-than-indices-size/47469/9 "2016-04-18T06:05:28Z")
Yes, that is correct. If you are constantly indexing into the cluster, merging of segments will continously happen in the background and the same record will end up in multiple segments over time, resulting in a repository that is considerably larger than the index size.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 10:58pm UTC](https://discuss.elastic.co/t/backup-repository-size-is-much-bigger-than-indices-size/47469/10 "2017-07-05T22:58:39Z")