# 39m tweets on two nodes requires 11tb on S3? **URL:** \ **Category:** Elasticsearch\ **Created:** [December 4, 2010, 4:58pm UTC](https://discuss.elastic.co/t/39m-tweets-on-two-nodes-requires-11tb-on-s3/3625 "2010-12-04T16:58:48Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Kenneth\_Loafman\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman_2/32/2224_2.png) [@Kenneth\_Loafman\_2](https://discuss.elastic.co/u/Kenneth_Loafman_2)\ **Post date:** [December 4, 2010, 4:58pm UTC](https://discuss.elastic.co/t/39m-tweets-on-two-nodes-requires-11tb-on-s3/3625/1 "2010-12-04T16:58:48Z")
Hi, ES 0.11.0-SNAPSHOT has been running solid for a couple of months with an S3 Gateway, but there's a serious problem. Status shows that there are 39m docs with a store size of 50.5 GB, see [https://gist.github.com/728313](https://gist.github.com/728313). An 'ls' of the contents of the nodes shows thousands of '\_\_\*.partNN' files, some dated a couple of months ago. A 'du' of the indices/twitter directory (2 nodes) shows 11 TB, quite a bit of waste space! What's the best way to compress and clean up this mess? Are old part files eligible for deletion? How do I keep this from happening again? ...Thanks, ...Ken ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 5, 2010, 3:34am UTC](https://discuss.elastic.co/t/39m-tweets-on-two-nodes-requires-11tb-on-s3/3625/2 "2010-12-05T03:34:20Z")
Each snapshot operation that happens should clean up at the end all files that are no longer referenced. The "part" files are broken down files into chunks to improve the write performance into S3 and work around the 5gb limit. Is there a chance that you can upgrade to 0.13.1? On Saturday, December 4, 2010 at 6:58 PM, Kenneth Loafman wrote: > Hi, > > ES 0.11.0-SNAPSHOT has been running solid for a couple of months with an S3 Gateway, but there's a serious problem. Status shows that there are 39m docs with a store size of 50.5 GB, see [twitter index status · GitHub](https://gist.github.com/728313). An 'ls' of the contents of the nodes shows thousands of '\_\_\*.partNN' files, some dated a couple of months ago. A 'du' of the indices/twitter directory (2 nodes) shows 11 TB, quite a bit of waste space! > > What's the best way to compress and clean up this mess? Are old part files eligible for deletion? How do I keep this from happening again? > > ...Thanks, > ...Ken ---
**Author:** ![Kenneth\_Loafman\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman_2/32/2224_2.png) [@Kenneth\_Loafman\_2](https://discuss.elastic.co/u/Kenneth_Loafman_2)\ **Post date:** [December 5, 2010, 3:50pm UTC](https://discuss.elastic.co/t/39m-tweets-on-two-nodes-requires-11tb-on-s3/3625/3 "2010-12-05T15:50:45Z")
It looks like the cleanups are not happening in 0.11 as they should. Will be upgrading to 0.13.1 today, in 'local' mode. This should clear up the problem, and since we have two nodes, we should be able to quiesce one, back it up, and bring it back online, at least that's the way I think the backup should be done. Any thoughts on that? On Sat, Dec 4, 2010 at 9:34 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Each snapshot operation that happens should clean up at the end all files > that are no longer referenced. The "part" files are broken down files into > chunks to improve the write performance into S3 and work around the 5gb > limit. Is there a chance that you can upgrade to 0.13.1? > > On Saturday, December 4, 2010 at 6:58 PM, Kenneth Loafman wrote: > > Hi, > > ES 0.11.0-SNAPSHOT has been running solid for a couple of months with an S3 > Gateway, but there's a serious problem. Status shows that there are 39m > docs with a store size of 50.5 GB, see [twitter index status · GitHub](https://gist.github.com/728313). An > 'ls' of the contents of the nodes shows thousands of '\_\_\*.partNN' files, > some dated a couple of months ago. A 'du' of the indices/twitter directory > (2 nodes) shows 11 TB, quite a bit of waste space! > > What's the best way to compress and clean up this mess? Are old part files > eligible for deletion? How do I keep this from happening again? > > ...Thanks, > ...Ken ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 5, 2010, 6:56pm UTC](https://discuss.elastic.co/t/39m-tweets-on-two-nodes-requires-11tb-on-s3/3625/4 "2010-12-05T18:56:52Z")
If you want to move to local gateway mode from shared (s3) gateway, you need to reindex the data. On Sunday, December 5, 2010 at 5:50 PM, Kenneth Loafman wrote: > It looks like the cleanups are not happening in 0.11 as they should. > > Will be upgrading to 0.13.1 today, in 'local' mode. This should clear up the problem, and since we have two nodes, we should be able to quiesce one, back it up, and bring it back online, at least that's the way I think the backup should be done. Any thoughts on that? > > On Sat, Dec 4, 2010 at 9:34 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > Each snapshot operation that happens should clean up at the end all files that are no longer referenced. The "part" files are broken down files into chunks to improve the write performance into S3 and work around the 5gb limit. Is there a chance that you can upgrade to 0.13.1? > > > > On Saturday, December 4, 2010 at 6:58 PM, Kenneth Loafman wrote: > > > > > Hi, > > > > > > ES 0.11.0-SNAPSHOT has been running solid for a couple of months with an S3 Gateway, but there's a serious problem. Status shows that there are 39m docs with a store size of 50.5 GB, see [twitter index status · GitHub](https://gist.github.com/728313). An 'ls' of the contents of the nodes shows thousands of '\_\_\*.partNN' files, some dated a couple of months ago. A 'du' of the indices/twitter directory (2 nodes) shows 11 TB, quite a bit of waste space! > > > > > > What's the best way to compress and clean up this mess? Are old part files eligible for deletion? How do I keep this from happening again? > > > > > > ...Thanks, > > > ...Ken ---
**Author:** ![Kenneth\_Loafman\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman_2/32/2224_2.png) [@Kenneth\_Loafman\_2](https://discuss.elastic.co/u/Kenneth_Loafman_2)\ **Post date:** [December 5, 2010, 7:03pm UTC](https://discuss.elastic.co/t/39m-tweets-on-two-nodes-requires-11tb-on-s3/3625/5 "2010-12-05T19:03:08Z")
Doing that now. I could not figure out how to clear out the old part files safely and wanted to get away from S3 anyway, so reindex it is. On Sun, Dec 5, 2010 at 12:56 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > If you want to move to local gateway mode from shared (s3) gateway, you > need to reindex the data. > > On Sunday, December 5, 2010 at 5:50 PM, Kenneth Loafman wrote: > > It looks like the cleanups are not happening in 0.11 as they should. > > Will be upgrading to 0.13.1 today, in 'local' mode. This should clear up > the problem, and since we have two nodes, we should be able to quiesce one, > back it up, and bring it back online, at least that's the way I think the > backup should be done. Any thoughts on that? > > On Sat, Dec 4, 2010 at 9:34 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > Each snapshot operation that happens should clean up at the end all files > that are no longer referenced. The "part" files are broken down files into > chunks to improve the write performance into S3 and work around the 5gb > limit. Is there a chance that you can upgrade to 0.13.1? > > On Saturday, December 4, 2010 at 6:58 PM, Kenneth Loafman wrote: > > Hi, > > ES 0.11.0-SNAPSHOT has been running solid for a couple of months with an S3 > Gateway, but there's a serious problem. Status shows that there are 39m > docs with a store size of 50.5 GB, see [twitter index status · GitHub](https://gist.github.com/728313). An > 'ls' of the contents of the nodes shows thousands of '\_\_\*.partNN' files, > some dated a couple of months ago. A 'du' of the indices/twitter directory > (2 nodes) shows 11 TB, quite a bit of waste space! > > What's the best way to compress and clean up this mess? Are old part files > eligible for deletion? How do I keep this from happening again? > > ...Thanks, > ...Ken ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 5, 2010, 7:04pm UTC](https://discuss.elastic.co/t/39m-tweets-on-two-nodes-requires-11tb-on-s3/3625/6 "2010-12-05T19:04:45Z")
Are you using EBS for the local storage? This should give you better IO perf (well, sometimes, depends how AWS woke up in the morning), and better long term persistency. On Sunday, December 5, 2010 at 9:03 PM, Kenneth Loafman wrote: > Doing that now. I could not figure out how to clear out the old part files safely and wanted to get away from S3 anyway, so reindex it is. > > On Sun, Dec 5, 2010 at 12:56 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > If you want to move to local gateway mode from shared (s3) gateway, you need to reindex the data. > > > > On Sunday, December 5, 2010 at 5:50 PM, Kenneth Loafman wrote: > > > > > It looks like the cleanups are not happening in 0.11 as they should. > > > > > > Will be upgrading to 0.13.1 today, in 'local' mode. This should clear up the problem, and since we have two nodes, we should be able to quiesce one, back it up, and bring it back online, at least that's the way I think the backup should be done. Any thoughts on that? > > > > > > On Sat, Dec 4, 2010 at 9:34 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > Each snapshot operation that happens should clean up at the end all files that are no longer referenced. The "part" files are broken down files into chunks to improve the write performance into S3 and work around the 5gb limit. Is there a chance that you can upgrade to 0.13.1? > > > > > > > > On Saturday, December 4, 2010 at 6:58 PM, Kenneth Loafman wrote: > > > > > > > > > Hi, > > > > > > > > > > ES 0.11.0-SNAPSHOT has been running solid for a couple of months with an S3 Gateway, but there's a serious problem. Status shows that there are 39m docs with a store size of 50.5 GB, see [twitter index status · GitHub](https://gist.github.com/728313). An 'ls' of the contents of the nodes shows thousands of '\_\_\*.partNN' files, some dated a couple of months ago. A 'du' of the indices/twitter directory (2 nodes) shows 11 TB, quite a bit of waste space! > > > > > > > > > > What's the best way to compress and clean up this mess? Are old part files eligible for deletion? How do I keep this from happening again? > > > > > > > > > > ...Thanks, > > > > > ...Ken ---
**Author:** ![Kenneth\_Loafman\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman_2/32/2224_2.png) [@Kenneth\_Loafman\_2](https://discuss.elastic.co/u/Kenneth_Loafman_2)\ **Post date:** [December 5, 2010, 8:04pm UTC](https://discuss.elastic.co/t/39m-tweets-on-two-nodes-requires-11tb-on-s3/3625/7 "2010-12-05T20:04:31Z")
No, just the local storage on the machines themselves. We're not hosted on Amazon and the transfer costs were getting horrendous, much more than the storage costs. On Sun, Dec 5, 2010 at 1:04 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Are you using EBS for the local storage? This should give you better IO > perf (well, sometimes, depends how AWS woke up in the morning), and better > long term persistency. > > On Sunday, December 5, 2010 at 9:03 PM, Kenneth Loafman wrote: > > Doing that now. I could not figure out how to clear out the old part files > safely and wanted to get away from S3 anyway, so reindex it is. > > On Sun, Dec 5, 2010 at 12:56 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > If you want to move to local gateway mode from shared (s3) gateway, you > need to reindex the data. > > On Sunday, December 5, 2010 at 5:50 PM, Kenneth Loafman wrote: > > It looks like the cleanups are not happening in 0.11 as they should. > > Will be upgrading to 0.13.1 today, in 'local' mode. This should clear up > the problem, and since we have two nodes, we should be able to quiesce one, > back it up, and bring it back online, at least that's the way I think the > backup should be done. Any thoughts on that? > > On Sat, Dec 4, 2010 at 9:34 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > Each snapshot operation that happens should clean up at the end all > files that are no longer referenced. The "part" files are broken down files > into chunks to improve the write performance into S3 and work around the 5gb > limit. Is there a chance that you can upgrade to 0.13.1? > > On Saturday, December 4, 2010 at 6:58 PM, Kenneth Loafman wrote: > > Hi, > > ES 0.11.0-SNAPSHOT has been running solid for a couple of months with an S3 > Gateway, but there's a serious problem. Status shows that there are 39m > docs with a store size of 50.5 GB, see [twitter index status · GitHub](https://gist.github.com/728313). An > 'ls' of the contents of the nodes shows thousands of '\_\_\*.partNN' files, > some dated a couple of months ago. A 'du' of the indices/twitter directory > (2 nodes) shows 11 TB, quite a bit of waste space! > > What's the best way to compress and clean up this mess? Are old part files > eligible for deletion? How do I keep this from happening again? > > ...Thanks, > ...Ken ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 5, 2010, 8:16pm UTC](https://discuss.elastic.co/t/39m-tweets-on-two-nodes-requires-11tb-on-s3/3625/8 "2010-12-05T20:16:38Z")
Right, I see. I agree, using s3 outside of amazon makes much less sense than using local gateway. On Sunday, December 5, 2010 at 10:04 PM, Kenneth Loafman wrote: > No, just the local storage on the machines themselves. We're not hosted on Amazon and the transfer costs were getting horrendous, much more than the storage costs. > > On Sun, Dec 5, 2010 at 1:04 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > Are you using EBS for the local storage? This should give you better IO perf (well, sometimes, depends how AWS woke up in the morning), and better long term persistency. > > > > On Sunday, December 5, 2010 at 9:03 PM, Kenneth Loafman wrote: > > > > > Doing that now. I could not figure out how to clear out the old part files safely and wanted to get away from S3 anyway, so reindex it is. > > > > > > On Sun, Dec 5, 2010 at 12:56 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > If you want to move to local gateway mode from shared (s3) gateway, you need to reindex the data. > > > > > > > > On Sunday, December 5, 2010 at 5:50 PM, Kenneth Loafman wrote: > > > > > > > > > It looks like the cleanups are not happening in 0.11 as they should. > > > > > > > > > > Will be upgrading to 0.13.1 today, in 'local' mode. This should clear up the problem, and since we have two nodes, we should be able to quiesce one, back it up, and bring it back online, at least that's the way I think the backup should be done. Any thoughts on that? > > > > > > > > > > On Sat, Dec 4, 2010 at 9:34 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > > > Each snapshot operation that happens should clean up at the end all files that are no longer referenced. The "part" files are broken down files into chunks to improve the write performance into S3 and work around the 5gb limit. Is there a chance that you can upgrade to 0.13.1? > > > > > > > > > > > > On Saturday, December 4, 2010 at 6:58 PM, Kenneth Loafman wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > ES 0.11.0-SNAPSHOT has been running solid for a couple of months with an S3 Gateway, but there's a serious problem. Status shows that there are 39m docs with a store size of 50.5 GB, see [twitter index status · GitHub](https://gist.github.com/728313). An 'ls' of the contents of the nodes shows thousands of '\_\_\*.partNN' files, some dated a couple of months ago. A 'du' of the indices/twitter directory (2 nodes) shows 11 TB, quite a bit of waste space! > > > > > > > > > > > > > > What's the best way to compress and clean up this mess? Are old part files eligible for deletion? How do I keep this from happening again? > > > > > > > > > > > > > > ...Thanks, > > > > > > > ...Ken ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:15am UTC](https://discuss.elastic.co/t/39m-tweets-on-two-nodes-requires-11tb-on-s3/3625/9 "2017-07-06T04:15:41Z")