# Breaking Change: Majore gateway refactoring, and improved throttling

**URL:** <https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245>\
**Category:** Elasticsearch\
**Created:** [August 23, 2010, 7:11pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245 "2010-08-23T19:11:05Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 23, 2010, 7:11pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245/1 "2010-08-23T19:11:05Z")

</div>

Hi,

```
I am going to push (pretty soon) a major rewrite of the gateway module

```

and improved throttling support. The gateway change is a breaking change,  
meaning that new version will not be able to recover from a 0.9 gateway. I  
will provide an upgrade script for file based gateway, s3 based gateway will  
require reindexing, though, potentially, that script can be adjusted to  
support it.

Let me explain some of the changes. The first is throttling support. In  
0.9, recoveries are being throttled on a specific node in order to reduce  
the load a that node. The throttling was done on the node level, after a  
shard has been allocated to it. Maintain the count of current recoveries is  
quite tricky because of the complexity of the recovery process. This has now  
been refactored into a better place, which is the actual  
allocation algorithm that runs and shuffles shards around.

The more interesting change is the gateway. There were several problems  
with how the gateway works today that were exposed by a user of  
elasticsearch that stores 4TB data (several indices, each with 10 shards and  
2 replicas, which sums it up to 12TB). This has uncovered some problems with  
the current design, specifically how md5 are computed (and the time it takes  
to compute them on the local storage on ec2), as well as other possibilities  
for gateway corruptions using this load. Of course, elasticsearch aim is to  
be able to store much more data than that, we are getting there... .

In general, the new implementation works (in spirit) in the same manner  
git works. Each snapshot is a commit point, that stores files in the gateway  
into an auto generated name, and finally, a commit point is written with the  
"directory" which maps between this pseudo name to physical name, and the  
size. The new design allows for more resiliency when it comes to corruption.  
It also allows for exciting future features like saving a commit point and  
restoring from it, or automatically create a commit point each day for the  
last 5 days and be able to rollback to a specific commit point.

The aim is to create a gateway storage that is going to be the final  
version, and resilient for future changes. It takes some time to get there,  
but once we are there, I can safely stand behind using elasticsearch as the  
main storage as well as releasing v1.0 (I think elasticsearch has enough  
features for 1.0, just the stability of the gateway is needed).

I would love for people to take this for a ride and check it out. The  
next version, as a result of that is going to be 0.10, and I will release it  
in the following days.

-shay.banon

---

<div class="post-metadata">

**Author:** ![talsalmona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/talsalmona/32/3316_2.png) [@talsalmona](https://discuss.elastic.co/u/talsalmona)\
**Post date:** [August 23, 2010, 7:21pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245/2 "2010-08-23T19:21:45Z")

</div>

Very nice feature indeed.  
Will this also allow for index level commit points?

Tal

On Aug 23, 10:11 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:

> Hi,
> 
> ```
> I am going to push (pretty soon) a major rewrite of the gateway module
> 
> ```
> 
> and improved throttling support. The gateway change is a breaking change,  
> meaning that new version will not be able to recover from a 0.9 gateway. I  
> will provide an upgrade script for file based gateway, s3 based gateway will  
> require reindexing, though, potentially, that script can be adjusted to  
> support it.
> 
> Let me explain some of the changes. The first is throttling support. In  
> 0.9, recoveries are being throttled on a specific node in order to reduce  
> the load a that node. The throttling was done on the node level, after a  
> shard has been allocated to it. Maintain the count of current recoveries is  
> quite tricky because of the complexity of the recovery process. This has now  
> been refactored into a better place, which is the actual  
> allocation algorithm that runs and shuffles shards around.
> 
> The more interesting change is the gateway. There were several problems  
> with how the gateway works today that were exposed by a user of  
> elasticsearch that stores 4TB data (several indices, each with 10 shards and  
> 2 replicas, which sums it up to 12TB). This has uncovered some problems with  
> the current design, specifically how md5 are computed (and the time it takes  
> to compute them on the local storage on ec2), as well as other possibilities  
> for gateway corruptions using this load. Of course, elasticsearch aim is to  
> be able to store much more data than that, we are getting there... .
> 
> In general, the new implementation works (in spirit) in the same manner  
> git works. Each snapshot is a commit point, that stores files in the gateway  
> into an auto generated name, and finally, a commit point is written with the  
> "directory" which maps between this pseudo name to physical name, and the  
> size. The new design allows for more resiliency when it comes to corruption.  
> It also allows for exciting future features like saving a commit point and  
> restoring from it, or automatically create a commit point each day for the  
> last 5 days and be able to rollback to a specific commit point.
> 
> The aim is to create a gateway storage that is going to be the final  
> version, and resilient for future changes. It takes some time to get there,  
> but once we are there, I can safely stand behind using elasticsearch as the  
> main storage as well as releasing v1.0 (I think elasticsearch has enough  
> features for 1.0, just the stability of the gateway is needed).
> 
> I would love for people to take this for a ride and check it out. The  
> next version, as a result of that is going to be 0.10, and I will release it  
> in the following days.
> 
> -shay.banon

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 23, 2010, 7:22pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245/3 "2010-08-23T19:22:59Z")

</div>

Yes, though there is a whole "workflow" level APIs and support for. But, the  
basics are there in the gateway.

-shay.banon

On Mon, Aug 23, 2010 at 10:21 PM, Tal [talsalmona@gmail.com](mailto:talsalmona@gmail.com) wrote:

> Very nice feature indeed.  
> Will this also allow for index level commit points?
> 
> Tal
> 
> On Aug 23, 10:11 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > Hi,
> > 
> > ```
> > I am going to push (pretty soon) a major rewrite of the gateway
> > 
> > ```
> 
> module
> 
> > and improved throttling support. The gateway change is a breaking change,  
> > meaning that new version will not be able to recover from a 0.9 gateway.  
> > I  
> > will provide an upgrade script for file based gateway, s3 based gateway  
> > will  
> > require reindexing, though, potentially, that script can be adjusted to  
> > support it.
> > 
> > Let me explain some of the changes. The first is throttling support.  
> > In  
> > 0.9, recoveries are being throttled on a specific node in order to reduce  
> > the load a that node. The throttling was done on the node level, after a  
> > shard has been allocated to it. Maintain the count of current recoveries  
> > is  
> > quite tricky because of the complexity of the recovery process. This has  
> > now  
> > been refactored into a better place, which is the actual  
> > allocation algorithm that runs and shuffles shards around.
> > 
> > The more interesting change is the gateway. There were several  
> > problems  
> > with how the gateway works today that were exposed by a user of  
> > elasticsearch that stores 4TB data (several indices, each with 10 shards  
> > and  
> > 2 replicas, which sums it up to 12TB). This has uncovered some problems  
> > with  
> > the current design, specifically how md5 are computed (and the time it  
> > takes  
> > to compute them on the local storage on ec2), as well as other  
> > possibilities  
> > for gateway corruptions using this load. Of course, elasticsearch aim is  
> > to  
> > be able to store much more data than that, we are getting there... .
> > 
> > In general, the new implementation works (in spirit) in the same  
> > manner  
> > git works. Each snapshot is a commit point, that stores files in the  
> > gateway  
> > into an auto generated name, and finally, a commit point is written with  
> > the  
> > "directory" which maps between this pseudo name to physical name, and the  
> > size. The new design allows for more resiliency when it comes to  
> > corruption.  
> > It also allows for exciting future features like saving a commit point  
> > and  
> > restoring from it, or automatically create a commit point each day for  
> > the  
> > last 5 days and be able to rollback to a specific commit point.
> > 
> > The aim is to create a gateway storage that is going to be the final  
> > version, and resilient for future changes. It takes some time to get  
> > there,  
> > but once we are there, I can safely stand behind using elasticsearch as  
> > the  
> > main storage as well as releasing v1.0 (I think elasticsearch has enough  
> > features for 1.0, just the stability of the gateway is needed).
> > 
> > I would love for people to take this for a ride and check it out. The  
> > next version, as a result of that is going to be 0.10, and I will release  
> > it  
> > in the following days.
> > 
> > -shay.banon

---

<div class="post-metadata">

**Author:** ![Grant\_Rodgers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grant_rodgers/32/3204_2.png) [@Grant\_Rodgers](https://discuss.elastic.co/u/Grant_Rodgers)\
**Post date:** [August 24, 2010, 7:23pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245/4 "2010-08-24T19:23:43Z")

</div>

I've been testing with the new stuff, seems cool. I saw this error  
while indexing, not sure if it's useful or not but I thought I'd pass  
it on:

[16:55:30,719][WARN][index.gateway] [Ankhi] [versions][1]  
failed to snapshot (scheduled)  
org.elasticsearch.index.gateway.IndexShardGatewaySnapshotFailedException:  
[versions][1] For input string: "\_1.swp"  
at  
org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.snapshot(BlobStoreIndexShardGateway.java:  
152)  
at org.elasticsearch.index.gateway.IndexShardGatewayService  
$2.snapshot(IndexShardGatewayService.java:232)  
at org.elasticsearch.index.gateway.IndexShardGatewayService  
$2.snapshot(IndexShardGatewayService.java:227)  
at  
org.elasticsearch.index.engine.robin.RobinEngine.snapshot(RobinEngine.java:  
426)  
at  
org.elasticsearch.index.shard.service.InternalIndexShard.snapshot(InternalIndexShard.java:  
372)  
at  
org.elasticsearch.index.gateway.IndexShardGatewayService.snapshot(IndexShardGatewayService.java:  
227)  
at org.elasticsearch.index.gateway.IndexShardGatewayService  
$SnapshotRunnable.run(IndexShardGatewayService.java:320)  
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:  
441)  
at java.util.concurrent.FutureTask  
$Sync.innerRunAndReset(FutureTask.java:317)  
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)  
at java.util.concurrent.ScheduledThreadPoolExecutor  
$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)  
at java.util.concurrent.ScheduledThreadPoolExecutor  
$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)  
at java.util.concurrent.ScheduledThreadPoolExecutor  
$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)  
at java.util.concurrent.ThreadPoolExecutor  
$Worker.runTask(ThreadPoolExecutor.java:886)  
at java.util.concurrent.ThreadPoolExecutor  
$Worker.run(ThreadPoolExecutor.java:908)  
at java.lang.Thread.run(Thread.java:619)  
Caused by: java.lang.NumberFormatException: For input string: "\_1.swp"  
at  
java.lang.NumberFormatException.forInputString(NumberFormatException.java:  
48)  
at java.lang.Long.parseLong(Long.java:410)  
at  
org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.findLatestFileNameGeneration(BlobStoreIndexShardGateway.java:  
809)  
at  
org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.doSnapshot(BlobStoreIndexShardGateway.java:  
169)  
at  
org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.snapshot(BlobStoreIndexShardGateway.java:  
142)  
... 15 more

On Aug 23, 12:22 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:

> Yes, though there is a whole "workflow" level APIs and support for. But, the  
> basics are there in the gateway.
> 
> -shay.banon
> 
> On Mon, Aug 23, 2010 at 10:21 PM, Tal [talsalm...@gmail.com](mailto:talsalm...@gmail.com) wrote:
> 
> > Very nice feature indeed.  
> > Will this also allow for index level commit points?
> 
> > Tal
> 
> > On Aug 23, 10:11 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > 
> > > Hi,
> 
> > > ```
> > > I am going to push (pretty soon) a major rewrite of the gateway
> > > 
> > > ```
> > 
> > module
> > 
> > > and improved throttling support. The gateway change is a breaking change,  
> > > meaning that new version will not be able to recover from a 0.9 gateway.  
> > > I  
> > > will provide an upgrade script for file based gateway, s3 based gateway  
> > > will  
> > > require reindexing, though, potentially, that script can be adjusted to  
> > > support it.
> 
> > > Let me explain some of the changes. The first is throttling support.  
> > > In  
> > > 0.9, recoveries are being throttled on a specific node in order to reduce  
> > > the load a that node. The throttling was done on the node level, after a  
> > > shard has been allocated to it. Maintain the count of current recoveries  
> > > is  
> > > quite tricky because of the complexity of the recovery process. This has  
> > > now  
> > > been refactored into a better place, which is the actual  
> > > allocation algorithm that runs and shuffles shards around.
> 
> > > The more interesting change is the gateway. There were several  
> > > problems  
> > > with how the gateway works today that were exposed by a user of  
> > > elasticsearch that stores 4TB data (several indices, each with 10 shards  
> > > and  
> > > 2 replicas, which sums it up to 12TB). This has uncovered some problems  
> > > with  
> > > the current design, specifically how md5 are computed (and the time it  
> > > takes  
> > > to compute them on the local storage on ec2), as well as other  
> > > possibilities  
> > > for gateway corruptions using this load. Of course, elasticsearch aim is  
> > > to  
> > > be able to store much more data than that, we are getting there... .
> 
> > > In general, the new implementation works (in spirit) in the same  
> > > manner  
> > > git works. Each snapshot is a commit point, that stores files in the  
> > > gateway  
> > > into an auto generated name, and finally, a commit point is written with  
> > > the  
> > > "directory" which maps between this pseudo name to physical name, and the  
> > > size. The new design allows for more resiliency when it comes to  
> > > corruption.  
> > > It also allows for exciting future features like saving a commit point  
> > > and  
> > > restoring from it, or automatically create a commit point each day for  
> > > the  
> > > last 5 days and be able to rollback to a specific commit point.
> 
> > > The aim is to create a gateway storage that is going to be the final  
> > > version, and resilient for future changes. It takes some time to get  
> > > there,  
> > > but once we are there, I can safely stand behind using elasticsearch as  
> > > the  
> > > main storage as well as releasing v1.0 (I think elasticsearch has enough  
> > > features for 1.0, just the stability of the gateway is needed).
> 
> > > I would love for people to take this for a ride and check it out. The  
> > > next version, as a result of that is going to be 0.10, and I will release  
> > > it  
> > > in the following days.
> 
> > > -shay.banon

---

<div class="post-metadata">

**Author:** ![Grant\_Rodgers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grant_rodgers/32/3204_2.png) [@Grant\_Rodgers](https://discuss.elastic.co/u/Grant_Rodgers)\
**Post date:** [August 24, 2010, 7:25pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245/5 "2010-08-24T19:25:08Z")

</div>

Oh btw that error was with 80c7135

On Aug 23, 12:22 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:

> Yes, though there is a whole "workflow" level APIs and support for. But, the  
> basics are there in the gateway.
> 
> -shay.banon
> 
> On Mon, Aug 23, 2010 at 10:21 PM, Tal [talsalm...@gmail.com](mailto:talsalm...@gmail.com) wrote:
> 
> > Very nice feature indeed.  
> > Will this also allow for index level commit points?
> 
> > Tal
> 
> > On Aug 23, 10:11 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > 
> > > Hi,
> 
> > > ```
> > > I am going to push (pretty soon) a major rewrite of the gateway
> > > 
> > > ```
> > 
> > module
> > 
> > > and improved throttling support. The gateway change is a breaking change,  
> > > meaning that new version will not be able to recover from a 0.9 gateway.  
> > > I  
> > > will provide an upgrade script for file based gateway, s3 based gateway  
> > > will  
> > > require reindexing, though, potentially, that script can be adjusted to  
> > > support it.
> 
> > > Let me explain some of the changes. The first is throttling support.  
> > > In  
> > > 0.9, recoveries are being throttled on a specific node in order to reduce  
> > > the load a that node. The throttling was done on the node level, after a  
> > > shard has been allocated to it. Maintain the count of current recoveries  
> > > is  
> > > quite tricky because of the complexity of the recovery process. This has  
> > > now  
> > > been refactored into a better place, which is the actual  
> > > allocation algorithm that runs and shuffles shards around.
> 
> > > The more interesting change is the gateway. There were several  
> > > problems  
> > > with how the gateway works today that were exposed by a user of  
> > > elasticsearch that stores 4TB data (several indices, each with 10 shards  
> > > and  
> > > 2 replicas, which sums it up to 12TB). This has uncovered some problems  
> > > with  
> > > the current design, specifically how md5 are computed (and the time it  
> > > takes  
> > > to compute them on the local storage on ec2), as well as other  
> > > possibilities  
> > > for gateway corruptions using this load. Of course, elasticsearch aim is  
> > > to  
> > > be able to store much more data than that, we are getting there... .
> 
> > > In general, the new implementation works (in spirit) in the same  
> > > manner  
> > > git works. Each snapshot is a commit point, that stores files in the  
> > > gateway  
> > > into an auto generated name, and finally, a commit point is written with  
> > > the  
> > > "directory" which maps between this pseudo name to physical name, and the  
> > > size. The new design allows for more resiliency when it comes to  
> > > corruption.  
> > > It also allows for exciting future features like saving a commit point  
> > > and  
> > > restoring from it, or automatically create a commit point each day for  
> > > the  
> > > last 5 days and be able to rollback to a specific commit point.
> 
> > > The aim is to create a gateway storage that is going to be the final  
> > > version, and resilient for future changes. It takes some time to get  
> > > there,  
> > > but once we are there, I can safely stand behind using elasticsearch as  
> > > the  
> > > main storage as well as releasing v1.0 (I think elasticsearch has enough  
> > > features for 1.0, just the stability of the gateway is needed).
> 
> > > I would love for people to take this for a ride and check it out. The  
> > > next version, as a result of that is going to be 0.10, and I will release  
> > > it  
> > > in the following days.
> 
> > > -shay.banon

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 24, 2010, 8:00pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245/6 "2010-08-24T20:00:10Z")

</div>

Strange, not sure how this file ended up in the gateway, I can't see where  
elasticsearch would write it. It only writes \_\_xxx files (no . something)  
and commit- files. I will fix it to ignore files that don't conform to the  
format, but we should try and understand where its coming from...

-shay.banon

On Tue, Aug 24, 2010 at 10:25 PM, Grant Rodgers [grantr@gmail.com](mailto:grantr@gmail.com) wrote:

> Oh btw that error was with 80c7135
> 
> On Aug 23, 12:22 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > Yes, though there is a whole "workflow" level APIs and support for. But,  
> > the  
> > basics are there in the gateway.
> > 
> > -shay.banon
> > 
> > On Mon, Aug 23, 2010 at 10:21 PM, Tal [talsalm...@gmail.com](mailto:talsalm...@gmail.com) wrote:
> > 
> > > Very nice feature indeed.  
> > > Will this also allow for index level commit points?
> > 
> > > Tal
> > 
> > > On Aug 23, 10:11 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > > 
> > > > Hi,
> > 
> > > > ```
> > > > I am going to push (pretty soon) a major rewrite of the gateway
> > > > 
> > > > ```
> > > 
> > > module
> > > 
> > > > and improved throttling support. The gateway change is a breaking  
> > > > change,  
> > > > meaning that new version will not be able to recover from a 0.9  
> > > > gateway.  
> > > > I  
> > > > will provide an upgrade script for file based gateway, s3 based  
> > > > gateway  
> > > > will  
> > > > require reindexing, though, potentially, that script can be adjusted  
> > > > to  
> > > > support it.
> > 
> > > > Let me explain some of the changes. The first is throttling  
> > > > support.  
> > > > In  
> > > > 0.9, recoveries are being throttled on a specific node in order to  
> > > > reduce  
> > > > the load a that node. The throttling was done on the node level,  
> > > > after a  
> > > > shard has been allocated to it. Maintain the count of current  
> > > > recoveries  
> > > > is  
> > > > quite tricky because of the complexity of the recovery process. This  
> > > > has  
> > > > now  
> > > > been refactored into a better place, which is the actual  
> > > > allocation algorithm that runs and shuffles shards around.
> > 
> > > > The more interesting change is the gateway. There were several  
> > > > problems  
> > > > with how the gateway works today that were exposed by a user of  
> > > > elasticsearch that stores 4TB data (several indices, each with 10  
> > > > shards  
> > > > and  
> > > > 2 replicas, which sums it up to 12TB). This has uncovered some  
> > > > problems  
> > > > with  
> > > > the current design, specifically how md5 are computed (and the time  
> > > > it  
> > > > takes  
> > > > to compute them on the local storage on ec2), as well as other  
> > > > possibilities  
> > > > for gateway corruptions using this load. Of course, elasticsearch aim  
> > > > is  
> > > > to  
> > > > be able to store much more data than that, we are getting there... .
> > 
> > > > In general, the new implementation works (in spirit) in the same  
> > > > manner  
> > > > git works. Each snapshot is a commit point, that stores files in the  
> > > > gateway  
> > > > into an auto generated name, and finally, a commit point is written  
> > > > with  
> > > > the  
> > > > "directory" which maps between this pseudo name to physical name, and  
> > > > the  
> > > > size. The new design allows for more resiliency when it comes to  
> > > > corruption.  
> > > > It also allows for exciting future features like saving a commit  
> > > > point  
> > > > and  
> > > > restoring from it, or automatically create a commit point each day  
> > > > for  
> > > > the  
> > > > last 5 days and be able to rollback to a specific commit point.
> > 
> > > > The aim is to create a gateway storage that is going to be the  
> > > > final  
> > > > version, and resilient for future changes. It takes some time to get  
> > > > there,  
> > > > but once we are there, I can safely stand behind using elasticsearch  
> > > > as  
> > > > the  
> > > > main storage as well as releasing v1.0 (I think elasticsearch has  
> > > > enough  
> > > > features for 1.0, just the stability of the gateway is needed).
> > 
> > > > I would love for people to take this for a ride and check it out.  
> > > > The  
> > > > next version, as a result of that is going to be 0.10, and I will  
> > > > release  
> > > > it  
> > > > in the following days.
> > 
> > > > -shay.banon

---

<div class="post-metadata">

**Author:** ![Grant\_Rodgers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grant_rodgers/32/3204_2.png) [@Grant\_Rodgers](https://discuss.elastic.co/u/Grant_Rodgers)\
**Post date:** [August 24, 2010, 8:12pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245/7 "2010-08-24T20:12:47Z")

</div>

Oh you know it was probably a vi swap file. I was taking a look at one  
of the commit logs, and it might have tried to snapshot while it was  
open.

I think you have committed another change since 80c7135 that ignores  
files elasticsearch didn't create. I'll build the latest head and try  
viewing a commit snapshot again.

On Aug 24, 1:00 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:

> Strange, not sure how this file ended up in the gateway, I can't see where  
> elasticsearch would write it. It only writes \_\_xxx files (no . something)  
> and commit- files. I will fix it to ignore files that don't conform to the  
> format, but we should try and understand where its coming from...
> 
> -shay.banon
> 
> On Tue, Aug 24, 2010 at 10:25 PM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com) wrote:
> 
> > Oh btw that error was with 80c7135
> 
> > On Aug 23, 12:22 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > 
> > > Yes, though there is a whole "workflow" level APIs and support for. But,  
> > > the  
> > > basics are there in the gateway.
> 
> > > -shay.banon
> 
> > > On Mon, Aug 23, 2010 at 10:21 PM, Tal [talsalm...@gmail.com](mailto:talsalm...@gmail.com) wrote:
> > > 
> > > > Very nice feature indeed.  
> > > > Will this also allow for index level commit points?
> 
> > > > Tal
> 
> > > > On Aug 23, 10:11 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > > > 
> > > > > Hi,
> 
> > > > > ```
> > > > > I am going to push (pretty soon) a major rewrite of the gateway
> > > > > 
> > > > > ```
> > > > 
> > > > module
> > > > 
> > > > > and improved throttling support. The gateway change is a breaking  
> > > > > change,  
> > > > > meaning that new version will not be able to recover from a 0.9  
> > > > > gateway.  
> > > > > I  
> > > > > will provide an upgrade script for file based gateway, s3 based  
> > > > > gateway  
> > > > > will  
> > > > > require reindexing, though, potentially, that script can be adjusted  
> > > > > to  
> > > > > support it.
> 
> > > > > Let me explain some of the changes. The first is throttling  
> > > > > support.  
> > > > > In  
> > > > > 0.9, recoveries are being throttled on a specific node in order to  
> > > > > reduce  
> > > > > the load a that node. The throttling was done on the node level,  
> > > > > after a  
> > > > > shard has been allocated to it. Maintain the count of current  
> > > > > recoveries  
> > > > > is  
> > > > > quite tricky because of the complexity of the recovery process. This  
> > > > > has  
> > > > > now  
> > > > > been refactored into a better place, which is the actual  
> > > > > allocation algorithm that runs and shuffles shards around.
> 
> > > > > The more interesting change is the gateway. There were several  
> > > > > problems  
> > > > > with how the gateway works today that were exposed by a user of  
> > > > > elasticsearch that stores 4TB data (several indices, each with 10  
> > > > > shards  
> > > > > and  
> > > > > 2 replicas, which sums it up to 12TB). This has uncovered some  
> > > > > problems  
> > > > > with  
> > > > > the current design, specifically how md5 are computed (and the time  
> > > > > it  
> > > > > takes  
> > > > > to compute them on the local storage on ec2), as well as other  
> > > > > possibilities  
> > > > > for gateway corruptions using this load. Of course, elasticsearch aim  
> > > > > is  
> > > > > to  
> > > > > be able to store much more data than that, we are getting there... .
> 
> > > > > In general, the new implementation works (in spirit) in the same  
> > > > > manner  
> > > > > git works. Each snapshot is a commit point, that stores files in the  
> > > > > gateway  
> > > > > into an auto generated name, and finally, a commit point is written  
> > > > > with  
> > > > > the  
> > > > > "directory" which maps between this pseudo name to physical name, and  
> > > > > the  
> > > > > size. The new design allows for more resiliency when it comes to  
> > > > > corruption.  
> > > > > It also allows for exciting future features like saving a commit  
> > > > > point  
> > > > > and  
> > > > > restoring from it, or automatically create a commit point each day  
> > > > > for  
> > > > > the  
> > > > > last 5 days and be able to rollback to a specific commit point.
> 
> > > > > The aim is to create a gateway storage that is going to be the  
> > > > > final  
> > > > > version, and resilient for future changes. It takes some time to get  
> > > > > there,  
> > > > > but once we are there, I can safely stand behind using elasticsearch  
> > > > > as  
> > > > > the  
> > > > > main storage as well as releasing v1.0 (I think elasticsearch has  
> > > > > enough  
> > > > > features for 1.0, just the stability of the gateway is needed).
> 
> > > > > I would love for people to take this for a ride and check it out.  
> > > > > The  
> > > > > next version, as a result of that is going to be 0.10, and I will  
> > > > > release  
> > > > > it  
> > > > > in the following days.
> 
> > > > > -shay.banon

---

<div class="post-metadata">

**Author:** ![Grant\_Rodgers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grant_rodgers/32/3204_2.png) [@Grant\_Rodgers](https://discuss.elastic.co/u/Grant_Rodgers)\
**Post date:** [August 24, 2010, 9:18pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245/8 "2010-08-24T21:18:26Z")

</div>

I think it's fixed in head; I didn't see this error again when trying  
the test below.

On Aug 24, 1:12 pm, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com) wrote:

> Oh you know it was probably a vi swap file. I was taking a look at one  
> of the commit logs, and it might have tried to snapshot while it was  
> open.
> 
> I think you have committed another change since 80c7135 that ignores  
> files elasticsearch didn't create. I'll build the latest head and try  
> viewing a commit snapshot again.
> 
> On Aug 24, 1:00 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > Strange, not sure how this file ended up in the gateway, I can't see where  
> > elasticsearch would write it. It only writes \_\_xxx files (no . something)  
> > and commit- files. I will fix it to ignore files that don't conform to the  
> > format, but we should try and understand where its coming from...
> 
> > -shay.banon
> 
> > On Tue, Aug 24, 2010 at 10:25 PM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com) wrote:
> > 
> > > Oh btw that error was with 80c7135
> 
> > > On Aug 23, 12:22 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > > 
> > > > Yes, though there is a whole "workflow" level APIs and support for. But,  
> > > > the  
> > > > basics are there in the gateway.
> 
> > > > -shay.banon
> 
> > > > On Mon, Aug 23, 2010 at 10:21 PM, Tal [talsalm...@gmail.com](mailto:talsalm...@gmail.com) wrote:
> > > > 
> > > > > Very nice feature indeed.  
> > > > > Will this also allow for index level commit points?
> 
> > > > > Tal
> 
> > > > > On Aug 23, 10:11 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > > > > 
> > > > > > Hi,
> 
> > > > > > ```
> > > > > > I am going to push (pretty soon) a major rewrite of the gateway
> > > > > > 
> > > > > > ```
> > > > > 
> > > > > module
> > > > > 
> > > > > > and improved throttling support. The gateway change is a breaking  
> > > > > > change,  
> > > > > > meaning that new version will not be able to recover from a 0.9  
> > > > > > gateway.  
> > > > > > I  
> > > > > > will provide an upgrade script for file based gateway, s3 based  
> > > > > > gateway  
> > > > > > will  
> > > > > > require reindexing, though, potentially, that script can be adjusted  
> > > > > > to  
> > > > > > support it.
> 
> > > > > > Let me explain some of the changes. The first is throttling  
> > > > > > support.  
> > > > > > In  
> > > > > > 0.9, recoveries are being throttled on a specific node in order to  
> > > > > > reduce  
> > > > > > the load a that node. The throttling was done on the node level,  
> > > > > > after a  
> > > > > > shard has been allocated to it. Maintain the count of current  
> > > > > > recoveries  
> > > > > > is  
> > > > > > quite tricky because of the complexity of the recovery process. This  
> > > > > > has  
> > > > > > now  
> > > > > > been refactored into a better place, which is the actual  
> > > > > > allocation algorithm that runs and shuffles shards around.
> 
> > > > > > The more interesting change is the gateway. There were several  
> > > > > > problems  
> > > > > > with how the gateway works today that were exposed by a user of  
> > > > > > elasticsearch that stores 4TB data (several indices, each with 10  
> > > > > > shards  
> > > > > > and  
> > > > > > 2 replicas, which sums it up to 12TB). This has uncovered some  
> > > > > > problems  
> > > > > > with  
> > > > > > the current design, specifically how md5 are computed (and the time  
> > > > > > it  
> > > > > > takes  
> > > > > > to compute them on the local storage on ec2), as well as other  
> > > > > > possibilities  
> > > > > > for gateway corruptions using this load. Of course, elasticsearch aim  
> > > > > > is  
> > > > > > to  
> > > > > > be able to store much more data than that, we are getting there... .
> 
> > > > > > In general, the new implementation works (in spirit) in the same  
> > > > > > manner  
> > > > > > git works. Each snapshot is a commit point, that stores files in the  
> > > > > > gateway  
> > > > > > into an auto generated name, and finally, a commit point is written  
> > > > > > with  
> > > > > > the  
> > > > > > "directory" which maps between this pseudo name to physical name, and  
> > > > > > the  
> > > > > > size. The new design allows for more resiliency when it comes to  
> > > > > > corruption.  
> > > > > > It also allows for exciting future features like saving a commit  
> > > > > > point  
> > > > > > and  
> > > > > > restoring from it, or automatically create a commit point each day  
> > > > > > for  
> > > > > > the  
> > > > > > last 5 days and be able to rollback to a specific commit point.
> 
> > > > > > The aim is to create a gateway storage that is going to be the  
> > > > > > final  
> > > > > > version, and resilient for future changes. It takes some time to get  
> > > > > > there,  
> > > > > > but once we are there, I can safely stand behind using elasticsearch  
> > > > > > as  
> > > > > > the  
> > > > > > main storage as well as releasing v1.0 (I think elasticsearch has  
> > > > > > enough  
> > > > > > features for 1.0, just the stability of the gateway is needed).
> 
> > > > > > I would love for people to take this for a ride and check it out.  
> > > > > > The  
> > > > > > next version, as a result of that is going to be 0.10, and I will  
> > > > > > release  
> > > > > > it  
> > > > > > in the following days.
> 
> > > > > > -shay.banon

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 24, 2010, 10:05pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245/9 "2010-08-24T22:05:18Z")

</div>

Great, thanks for validating that. Did not expect someone to open vi on the  
gateway 😉

On Wed, Aug 25, 2010 at 12:18 AM, Grant Rodgers [grantr@gmail.com](mailto:grantr@gmail.com) wrote:

> I think it's fixed in head; I didn't see this error again when trying  
> the test below.
> 
> On Aug 24, 1:12 pm, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com) wrote:
> 
> > Oh you know it was probably a vi swap file. I was taking a look at one  
> > of the commit logs, and it might have tried to snapshot while it was  
> > open.
> > 
> > I think you have committed another change since 80c7135 that ignores  
> > files elasticsearch didn't create. I'll build the latest head and try  
> > viewing a commit snapshot again.
> > 
> > On Aug 24, 1:00 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > 
> > > Strange, not sure how this file ended up in the gateway, I can't see  
> > > where  
> > > elasticsearch would write it. It only writes \_\_xxx files (no .  
> > > something)  
> > > and commit- files. I will fix it to ignore files that don't conform to  
> > > the  
> > > format, but we should try and understand where its coming from...
> > 
> > > -shay.banon
> > 
> > > On Tue, Aug 24, 2010 at 10:25 PM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com)  
> > > wrote:
> > > 
> > > > Oh btw that error was with 80c7135
> > 
> > > > On Aug 23, 12:22 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)  
> > > > wrote:
> > > > 
> > > > > Yes, though there is a whole "workflow" level APIs and support for.  
> > > > > But,  
> > > > > the  
> > > > > basics are there in the gateway.
> > 
> > > > > -shay.banon
> > 
> > > > > On Mon, Aug 23, 2010 at 10:21 PM, Tal [talsalm...@gmail.com](mailto:talsalm...@gmail.com)  
> > > > > wrote:
> > > > > 
> > > > > > Very nice feature indeed.  
> > > > > > Will this also allow for index level commit points?
> > 
> > > > > > Tal
> > 
> > > > > > On Aug 23, 10:11 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)  
> > > > > > wrote:
> > > > > > 
> > > > > > > Hi,
> > 
> > > > > > > ```
> > > > > > > I am going to push (pretty soon) a major rewrite of the
> > > > > > > 
> > > > > > > ```
> 
> gateway
> 
> > > > > > module
> > > > > > 
> > > > > > > and improved throttling support. The gateway change is a  
> > > > > > > breaking  
> > > > > > > change,  
> > > > > > > meaning that new version will not be able to recover from a 0.9  
> > > > > > > gateway.  
> > > > > > > I  
> > > > > > > will provide an upgrade script for file based gateway, s3 based  
> > > > > > > gateway  
> > > > > > > will  
> > > > > > > require reindexing, though, potentially, that script can be  
> > > > > > > adjusted  
> > > > > > > to  
> > > > > > > support it.
> > 
> > > > > > > Let me explain some of the changes. The first is throttling  
> > > > > > > support.  
> > > > > > > In  
> > > > > > > 0.9, recoveries are being throttled on a specific node in order  
> > > > > > > to  
> > > > > > > reduce  
> > > > > > > the load a that node. The throttling was done on the node  
> > > > > > > level,  
> > > > > > > after a  
> > > > > > > shard has been allocated to it. Maintain the count of current  
> > > > > > > recoveries  
> > > > > > > is  
> > > > > > > quite tricky because of the complexity of the recovery process.  
> > > > > > > This  
> > > > > > > has  
> > > > > > > now  
> > > > > > > been refactored into a better place, which is the actual  
> > > > > > > allocation algorithm that runs and shuffles shards around.
> > 
> > > > > > > The more interesting change is the gateway. There were  
> > > > > > > several  
> > > > > > > problems  
> > > > > > > with how the gateway works today that were exposed by a user of  
> > > > > > > elasticsearch that stores 4TB data (several indices, each with  
> > > > > > > 10  
> > > > > > > shards  
> > > > > > > and  
> > > > > > > 2 replicas, which sums it up to 12TB). This has uncovered some  
> > > > > > > problems  
> > > > > > > with  
> > > > > > > the current design, specifically how md5 are computed (and the  
> > > > > > > time  
> > > > > > > it  
> > > > > > > takes  
> > > > > > > to compute them on the local storage on ec2), as well as other  
> > > > > > > possibilities  
> > > > > > > for gateway corruptions using this load. Of course,  
> > > > > > > elasticsearch aim  
> > > > > > > is  
> > > > > > > to  
> > > > > > > be able to store much more data than that, we are getting  
> > > > > > > there... .
> > 
> > > > > > > In general, the new implementation works (in spirit) in the  
> > > > > > > same  
> > > > > > > manner  
> > > > > > > git works. Each snapshot is a commit point, that stores files  
> > > > > > > in the  
> > > > > > > gateway  
> > > > > > > into an auto generated name, and finally, a commit point is  
> > > > > > > written  
> > > > > > > with  
> > > > > > > the  
> > > > > > > "directory" which maps between this pseudo name to physical  
> > > > > > > name, and  
> > > > > > > the  
> > > > > > > size. The new design allows for more resiliency when it comes  
> > > > > > > to  
> > > > > > > corruption.  
> > > > > > > It also allows for exciting future features like saving a  
> > > > > > > commit  
> > > > > > > point  
> > > > > > > and  
> > > > > > > restoring from it, or automatically create a commit point each  
> > > > > > > day  
> > > > > > > for  
> > > > > > > the  
> > > > > > > last 5 days and be able to rollback to a specific commit point.
> > 
> > > > > > > The aim is to create a gateway storage that is going to be  
> > > > > > > the  
> > > > > > > final  
> > > > > > > version, and resilient for future changes. It takes some time  
> > > > > > > to get  
> > > > > > > there,  
> > > > > > > but once we are there, I can safely stand behind using  
> > > > > > > elasticsearch  
> > > > > > > as  
> > > > > > > the  
> > > > > > > main storage as well as releasing v1.0 (I think elasticsearch  
> > > > > > > has  
> > > > > > > enough  
> > > > > > > features for 1.0, just the stability of the gateway is needed).
> > 
> > > > > > > I would love for people to take this for a ride and check it  
> > > > > > > out.  
> > > > > > > The  
> > > > > > > next version, as a result of that is going to be 0.10, and I  
> > > > > > > will  
> > > > > > > release  
> > > > > > > it  
> > > > > > > in the following days.
> > 
> > > > > > > -shay.banon

---

<div class="post-metadata">

**Author:** ![Grant\_Rodgers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grant_rodgers/32/3204_2.png) [@Grant\_Rodgers](https://discuss.elastic.co/u/Grant_Rodgers)\
**Post date:** [August 24, 2010, 10:32pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245/10 "2010-08-24T22:32:47Z")

</div>

I was very curious to see what the file format was!

On Aug 24, 3:05 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:

> Great, thanks for validating that. Did not expect someone to open vi on the  
> gateway 😉
> 
> On Wed, Aug 25, 2010 at 12:18 AM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com) wrote:
> 
> > I think it's fixed in head; I didn't see this error again when trying  
> > the test below.
> 
> > On Aug 24, 1:12 pm, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com) wrote:
> > 
> > > Oh you know it was probably a vi swap file. I was taking a look at one  
> > > of the commit logs, and it might have tried to snapshot while it was  
> > > open.
> 
> > > I think you have committed another change since 80c7135 that ignores  
> > > files elasticsearch didn't create. I'll build the latest head and try  
> > > viewing a commit snapshot again.
> 
> > > On Aug 24, 1:00 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > > > Strange, not sure how this file ended up in the gateway, I can't see  
> > > > where  
> > > > elasticsearch would write it. It only writes \_\_xxx files (no .  
> > > > something)  
> > > > and commit- files. I will fix it to ignore files that don't conform to  
> > > > the  
> > > > format, but we should try and understand where its coming from...
> 
> > > > -shay.banon
> 
> > > > On Tue, Aug 24, 2010 at 10:25 PM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com)  
> > > > wrote:
> > > > 
> > > > > Oh btw that error was with 80c7135
> 
> > > > > On Aug 23, 12:22 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)  
> > > > > wrote:
> > > > > 
> > > > > > Yes, though there is a whole "workflow" level APIs and support for.  
> > > > > > But,  
> > > > > > the  
> > > > > > basics are there in the gateway.
> 
> > > > > > -shay.banon
> 
> > > > > > On Mon, Aug 23, 2010 at 10:21 PM, Tal [talsalm...@gmail.com](mailto:talsalm...@gmail.com)  
> > > > > > wrote:
> > > > > > 
> > > > > > > Very nice feature indeed.  
> > > > > > > Will this also allow for index level commit points?
> 
> > > > > > > Tal
> 
> > > > > > > On Aug 23, 10:11 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)  
> > > > > > > wrote:
> > > > > > > 
> > > > > > > > Hi,
> 
> > > > > > > > ```
> > > > > > > > I am going to push (pretty soon) a major rewrite of the
> > > > > > > > 
> > > > > > > > ```
> > 
> > gateway
> > 
> > > > > > > module
> > > > > > > 
> > > > > > > > and improved throttling support. The gateway change is a  
> > > > > > > > breaking  
> > > > > > > > change,  
> > > > > > > > meaning that new version will not be able to recover from a 0.9  
> > > > > > > > gateway.  
> > > > > > > > I  
> > > > > > > > will provide an upgrade script for file based gateway, s3 based  
> > > > > > > > gateway  
> > > > > > > > will  
> > > > > > > > require reindexing, though, potentially, that script can be  
> > > > > > > > adjusted  
> > > > > > > > to  
> > > > > > > > support it.
> 
> > > > > > > > Let me explain some of the changes. The first is throttling  
> > > > > > > > support.  
> > > > > > > > In  
> > > > > > > > 0.9, recoveries are being throttled on a specific node in order  
> > > > > > > > to  
> > > > > > > > reduce  
> > > > > > > > the load a that node. The throttling was done on the node  
> > > > > > > > level,  
> > > > > > > > after a  
> > > > > > > > shard has been allocated to it. Maintain the count of current  
> > > > > > > > recoveries  
> > > > > > > > is  
> > > > > > > > quite tricky because of the complexity of the recovery process.  
> > > > > > > > This  
> > > > > > > > has  
> > > > > > > > now  
> > > > > > > > been refactored into a better place, which is the actual  
> > > > > > > > allocation algorithm that runs and shuffles shards around.
> 
> > > > > > > > The more interesting change is the gateway. There were  
> > > > > > > > several  
> > > > > > > > problems  
> > > > > > > > with how the gateway works today that were exposed by a user of  
> > > > > > > > elasticsearch that stores 4TB data (several indices, each with  
> > > > > > > > 10  
> > > > > > > > shards  
> > > > > > > > and  
> > > > > > > > 2 replicas, which sums it up to 12TB). This has uncovered some  
> > > > > > > > problems  
> > > > > > > > with  
> > > > > > > > the current design, specifically how md5 are computed (and the  
> > > > > > > > time  
> > > > > > > > it  
> > > > > > > > takes  
> > > > > > > > to compute them on the local storage on ec2), as well as other  
> > > > > > > > possibilities  
> > > > > > > > for gateway corruptions using this load. Of course,  
> > > > > > > > elasticsearch aim  
> > > > > > > > is  
> > > > > > > > to  
> > > > > > > > be able to store much more data than that, we are getting  
> > > > > > > > there... .
> 
> > > > > > > > In general, the new implementation works (in spirit) in the  
> > > > > > > > same  
> > > > > > > > manner  
> > > > > > > > git works. Each snapshot is a commit point, that stores files  
> > > > > > > > in the  
> > > > > > > > gateway  
> > > > > > > > into an auto generated name, and finally, a commit point is  
> > > > > > > > written  
> > > > > > > > with  
> > > > > > > > the  
> > > > > > > > "directory" which maps between this pseudo name to physical  
> > > > > > > > name, and  
> > > > > > > > the  
> > > > > > > > size. The new design allows for more resiliency when it comes  
> > > > > > > > to  
> > > > > > > > corruption.  
> > > > > > > > It also allows for exciting future features like saving a  
> > > > > > > > commit  
> > > > > > > > point  
> > > > > > > > and  
> > > > > > > > restoring from it, or automatically create a commit point each  
> > > > > > > > day  
> > > > > > > > for  
> > > > > > > > the  
> > > > > > > > last 5 days and be able to rollback to a specific commit point.
> 
> > > > > > > > The aim is to create a gateway storage that is going to be  
> > > > > > > > the  
> > > > > > > > final  
> > > > > > > > version, and resilient for future changes. It takes some time  
> > > > > > > > to get  
> > > > > > > > there,  
> > > > > > > > but once we are there, I can safely stand behind using  
> > > > > > > > elasticsearch  
> > > > > > > > as  
> > > > > > > > the  
> > > > > > > > main storage as well as releasing v1.0 (I think elasticsearch  
> > > > > > > > has  
> > > > > > > > enough  
> > > > > > > > features for 1.0, just the stability of the gateway is needed).
> 
> > > > > > > > I would love for people to take this for a ride and check it  
> > > > > > > > out.  
> > > > > > > > The  
> > > > > > > > next version, as a result of that is going to be 0.10, and I  
> > > > > > > > will  
> > > > > > > > release  
> > > > > > > > it  
> > > > > > > > in the following days.
> 
> > > > > > > > -shay.banon

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 24, 2010, 10:34pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245/11 "2010-08-24T22:34:58Z")

</div>

Indeed. All the \_\_xxx fils are binary files (either index files or  
transaction log parts). The commit-N is a json file that provides meta  
information on the commit point.

On Wed, Aug 25, 2010 at 1:32 AM, Grant Rodgers [grantr@gmail.com](mailto:grantr@gmail.com) wrote:

> I was very curious to see what the file format was!
> 
> On Aug 24, 3:05 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > Great, thanks for validating that. Did not expect someone to open vi on  
> > the  
> > gateway 😉
> > 
> > On Wed, Aug 25, 2010 at 12:18 AM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com)  
> > wrote:
> > 
> > > I think it's fixed in head; I didn't see this error again when trying  
> > > the test below.
> > 
> > > On Aug 24, 1:12 pm, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com) wrote:
> > > 
> > > > Oh you know it was probably a vi swap file. I was taking a look at  
> > > > one  
> > > > of the commit logs, and it might have tried to snapshot while it was  
> > > > open.
> > 
> > > > I think you have committed another change since 80c7135 that ignores  
> > > > files elasticsearch didn't create. I'll build the latest head and try  
> > > > viewing a commit snapshot again.
> > 
> > > > On Aug 24, 1:00 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > 
> > > > > Strange, not sure how this file ended up in the gateway, I can't  
> > > > > see  
> > > > > where  
> > > > > elasticsearch would write it. It only writes \_\_xxx files (no .  
> > > > > something)  
> > > > > and commit- files. I will fix it to ignore files that don't conform  
> > > > > to  
> > > > > the  
> > > > > format, but we should try and understand where its coming from...
> > 
> > > > > -shay.banon
> > 
> > > > > On Tue, Aug 24, 2010 at 10:25 PM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com)  
> > > > > wrote:
> > > > > 
> > > > > > Oh btw that error was with 80c7135
> > 
> > > > > > On Aug 23, 12:22 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)  
> > > > > > wrote:
> > > > > > 
> > > > > > > Yes, though there is a whole "workflow" level APIs and support  
> > > > > > > for.  
> > > > > > > But,  
> > > > > > > the  
> > > > > > > basics are there in the gateway.
> > 
> > > > > > > -shay.banon
> > 
> > > > > > > On Mon, Aug 23, 2010 at 10:21 PM, Tal [talsalm...@gmail.com](mailto:talsalm...@gmail.com)  
> > > > > > > wrote:
> > > > > > > 
> > > > > > > > Very nice feature indeed.  
> > > > > > > > Will this also allow for index level commit points?
> > 
> > > > > > > > Tal
> > 
> > > > > > > > On Aug 23, 10:11 pm, Shay Banon \<  
> > > > > > > > [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)\>  
> > > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > Hi,
> > 
> > > > > > > > > ```
> > > > > > > > > I am going to push (pretty soon) a major rewrite of the
> > > > > > > > > 
> > > > > > > > > ```
> > > 
> > > gateway
> > > 
> > > > > > > > module
> > > > > > > > 
> > > > > > > > > and improved throttling support. The gateway change is a  
> > > > > > > > > breaking  
> > > > > > > > > change,  
> > > > > > > > > meaning that new version will not be able to recover from a  
> > > > > > > > > 0.9  
> > > > > > > > > gateway.  
> > > > > > > > > I  
> > > > > > > > > will provide an upgrade script for file based gateway, s3  
> > > > > > > > > based  
> > > > > > > > > gateway  
> > > > > > > > > will  
> > > > > > > > > require reindexing, though, potentially, that script can be  
> > > > > > > > > adjusted  
> > > > > > > > > to  
> > > > > > > > > support it.
> > 
> > > > > > > > > Let me explain some of the changes. The first is  
> > > > > > > > > throttling  
> > > > > > > > > support.  
> > > > > > > > > In  
> > > > > > > > > 0.9, recoveries are being throttled on a specific node in  
> > > > > > > > > order  
> > > > > > > > > to  
> > > > > > > > > reduce  
> > > > > > > > > the load a that node. The throttling was done on the node  
> > > > > > > > > level,  
> > > > > > > > > after a  
> > > > > > > > > shard has been allocated to it. Maintain the count of  
> > > > > > > > > current  
> > > > > > > > > recoveries  
> > > > > > > > > is  
> > > > > > > > > quite tricky because of the complexity of the recovery  
> > > > > > > > > process.  
> > > > > > > > > This  
> > > > > > > > > has  
> > > > > > > > > now  
> > > > > > > > > been refactored into a better place, which is the actual  
> > > > > > > > > allocation algorithm that runs and shuffles shards around.
> > 
> > > > > > > > > The more interesting change is the gateway. There were  
> > > > > > > > > several  
> > > > > > > > > problems  
> > > > > > > > > with how the gateway works today that were exposed by a  
> > > > > > > > > user of  
> > > > > > > > > elasticsearch that stores 4TB data (several indices, each  
> > > > > > > > > with  
> > > > > > > > > 10  
> > > > > > > > > shards  
> > > > > > > > > and  
> > > > > > > > > 2 replicas, which sums it up to 12TB). This has uncovered  
> > > > > > > > > some  
> > > > > > > > > problems  
> > > > > > > > > with  
> > > > > > > > > the current design, specifically how md5 are computed (and  
> > > > > > > > > the  
> > > > > > > > > time  
> > > > > > > > > it  
> > > > > > > > > takes  
> > > > > > > > > to compute them on the local storage on ec2), as well as  
> > > > > > > > > other  
> > > > > > > > > possibilities  
> > > > > > > > > for gateway corruptions using this load. Of course,  
> > > > > > > > > elasticsearch aim  
> > > > > > > > > is  
> > > > > > > > > to  
> > > > > > > > > be able to store much more data than that, we are getting  
> > > > > > > > > there... .
> > 
> > > > > > > > > In general, the new implementation works (in spirit) in  
> > > > > > > > > the  
> > > > > > > > > same  
> > > > > > > > > manner  
> > > > > > > > > git works. Each snapshot is a commit point, that stores  
> > > > > > > > > files  
> > > > > > > > > in the  
> > > > > > > > > gateway  
> > > > > > > > > into an auto generated name, and finally, a commit point is  
> > > > > > > > > written  
> > > > > > > > > with  
> > > > > > > > > the  
> > > > > > > > > "directory" which maps between this pseudo name to physical  
> > > > > > > > > name, and  
> > > > > > > > > the  
> > > > > > > > > size. The new design allows for more resiliency when it  
> > > > > > > > > comes  
> > > > > > > > > to  
> > > > > > > > > corruption.  
> > > > > > > > > It also allows for exciting future features like saving a  
> > > > > > > > > commit  
> > > > > > > > > point  
> > > > > > > > > and  
> > > > > > > > > restoring from it, or automatically create a commit point  
> > > > > > > > > each  
> > > > > > > > > day  
> > > > > > > > > for  
> > > > > > > > > the  
> > > > > > > > > last 5 days and be able to rollback to a specific commit  
> > > > > > > > > point.
> > 
> > > > > > > > > The aim is to create a gateway storage that is going to  
> > > > > > > > > be  
> > > > > > > > > the  
> > > > > > > > > final  
> > > > > > > > > version, and resilient for future changes. It takes some  
> > > > > > > > > time  
> > > > > > > > > to get  
> > > > > > > > > there,  
> > > > > > > > > but once we are there, I can safely stand behind using  
> > > > > > > > > elasticsearch  
> > > > > > > > > as  
> > > > > > > > > the  
> > > > > > > > > main storage as well as releasing v1.0 (I think  
> > > > > > > > > elasticsearch  
> > > > > > > > > has  
> > > > > > > > > enough  
> > > > > > > > > features for 1.0, just the stability of the gateway is  
> > > > > > > > > needed).
> > 
> > > > > > > > > I would love for people to take this for a ride and  
> > > > > > > > > check it  
> > > > > > > > > out.  
> > > > > > > > > The  
> > > > > > > > > next version, as a result of that is going to be 0.10, and  
> > > > > > > > > I  
> > > > > > > > > will  
> > > > > > > > > release  
> > > > > > > > > it  
> > > > > > > > > in the following days.
> > 
> > > > > > > > > -shay.banon

---

<div class="post-metadata">

**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\
**Post date:** [August 25, 2010, 5:46pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245/12 "2010-08-25T17:46:11Z")

</div>

Hi Shay,  
Just curious, is the conversion script for file based gateway is  
available? I haven't searched through GIT, so maybe it is there?

Spent the weekend building up 20mil docs and would prefer not to  
repeat this.

Thanks,  
Paul

On Aug 24, 4:34 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:

> Indeed. All the \_\_xxx fils are binary files (either index files or  
> transaction log parts). The commit-N is a json file that provides meta  
> information on the commit point.
> 
> On Wed, Aug 25, 2010 at 1:32 AM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com) wrote:
> 
> > I was very curious to see what the file format was!
> 
> > On Aug 24, 3:05 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > 
> > > Great, thanks for validating that. Did not expect someone to open vi on  
> > > the  
> > > gateway 😉
> 
> > > On Wed, Aug 25, 2010 at 12:18 AM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com)  
> > > wrote:
> > > 
> > > > I think it's fixed in head; I didn't see this error again when trying  
> > > > the test below.
> 
> > > > On Aug 24, 1:12 pm, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com) wrote:
> > > > 
> > > > > Oh you know it was probably a vi swap file. I was taking a look at  
> > > > > one  
> > > > > of the commit logs, and it might have tried to snapshot while it was  
> > > > > open.
> 
> > > > > I think you have committed another change since 80c7135 that ignores  
> > > > > files elasticsearch didn't create. I'll build the latest head and try  
> > > > > viewing a commit snapshot again.
> 
> > > > > On Aug 24, 1:00 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > > > > > Strange, not sure how this file ended up in the gateway, I can't  
> > > > > > see  
> > > > > > where  
> > > > > > elasticsearch would write it. It only writes \_\_xxx files (no .  
> > > > > > something)  
> > > > > > and commit- files. I will fix it to ignore files that don't conform  
> > > > > > to  
> > > > > > the  
> > > > > > format, but we should try and understand where its coming from...
> 
> > > > > > -shay.banon
> 
> > > > > > On Tue, Aug 24, 2010 at 10:25 PM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com)  
> > > > > > wrote:
> > > > > > 
> > > > > > > Oh btw that error was with 80c7135
> 
> > > > > > > On Aug 23, 12:22 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)  
> > > > > > > wrote:
> > > > > > > 
> > > > > > > > Yes, though there is a whole "workflow" level APIs and support  
> > > > > > > > for.  
> > > > > > > > But,  
> > > > > > > > the  
> > > > > > > > basics are there in the gateway.
> 
> > > > > > > > -shay.banon
> 
> > > > > > > > On Mon, Aug 23, 2010 at 10:21 PM, Tal [talsalm...@gmail.com](mailto:talsalm...@gmail.com)  
> > > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > Very nice feature indeed.  
> > > > > > > > > Will this also allow for index level commit points?
> 
> > > > > > > > > Tal
> 
> > > > > > > > > On Aug 23, 10:11 pm, Shay Banon \<  
> > > > > > > > > [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)\>  
> > > > > > > > > wrote:
> > > > > > > > > 
> > > > > > > > > > Hi,
> 
> > > > > > > > > > ```
> > > > > > > > > > I am going to push (pretty soon) a major rewrite of the
> > > > > > > > > > 
> > > > > > > > > > ```
> > > > 
> > > > gateway
> > > > 
> > > > > > > > > module
> > > > > > > > > 
> > > > > > > > > > and improved throttling support. The gateway change is a  
> > > > > > > > > > breaking  
> > > > > > > > > > change,  
> > > > > > > > > > meaning that new version will not be able to recover from a  
> > > > > > > > > > 0.9  
> > > > > > > > > > gateway.  
> > > > > > > > > > I  
> > > > > > > > > > will provide an upgrade script for file based gateway, s3  
> > > > > > > > > > based  
> > > > > > > > > > gateway  
> > > > > > > > > > will  
> > > > > > > > > > require reindexing, though, potentially, that script can be  
> > > > > > > > > > adjusted  
> > > > > > > > > > to  
> > > > > > > > > > support it.
> 
> > > > > > > > > > Let me explain some of the changes. The first is  
> > > > > > > > > > throttling  
> > > > > > > > > > support.  
> > > > > > > > > > In  
> > > > > > > > > > 0.9, recoveries are being throttled on a specific node in  
> > > > > > > > > > order  
> > > > > > > > > > to  
> > > > > > > > > > reduce  
> > > > > > > > > > the load a that node. The throttling was done on the node  
> > > > > > > > > > level,  
> > > > > > > > > > after a  
> > > > > > > > > > shard has been allocated to it. Maintain the count of  
> > > > > > > > > > current  
> > > > > > > > > > recoveries  
> > > > > > > > > > is  
> > > > > > > > > > quite tricky because of the complexity of the recovery  
> > > > > > > > > > process.  
> > > > > > > > > > This  
> > > > > > > > > > has  
> > > > > > > > > > now  
> > > > > > > > > > been refactored into a better place, which is the actual  
> > > > > > > > > > allocation algorithm that runs and shuffles shards around.
> 
> > > > > > > > > > The more interesting change is the gateway. There were  
> > > > > > > > > > several  
> > > > > > > > > > problems  
> > > > > > > > > > with how the gateway works today that were exposed by a  
> > > > > > > > > > user of  
> > > > > > > > > > elasticsearch that stores 4TB data (several indices, each  
> > > > > > > > > > with  
> > > > > > > > > > 10  
> > > > > > > > > > shards  
> > > > > > > > > > and  
> > > > > > > > > > 2 replicas, which sums it up to 12TB). This has uncovered  
> > > > > > > > > > some  
> > > > > > > > > > problems  
> > > > > > > > > > with  
> > > > > > > > > > the current design, specifically how md5 are computed (and  
> > > > > > > > > > the  
> > > > > > > > > > time  
> > > > > > > > > > it  
> > > > > > > > > > takes  
> > > > > > > > > > to compute them on the local storage on ec2), as well as  
> > > > > > > > > > other  
> > > > > > > > > > possibilities  
> > > > > > > > > > for gateway corruptions using this load. Of course,  
> > > > > > > > > > elasticsearch aim  
> > > > > > > > > > is  
> > > > > > > > > > to  
> > > > > > > > > > be able to store much more data than that, we are getting  
> > > > > > > > > > there... .
> 
> > > > > > > > > > In general, the new implementation works (in spirit) in  
> > > > > > > > > > the  
> > > > > > > > > > same  
> > > > > > > > > > manner  
> > > > > > > > > > git works. Each snapshot is a commit point, that stores  
> > > > > > > > > > files  
> > > > > > > > > > in the  
> > > > > > > > > > gateway  
> > > > > > > > > > into an auto generated name, and finally, a commit point is  
> > > > > > > > > > written  
> > > > > > > > > > with  
> > > > > > > > > > the  
> > > > > > > > > > "directory" which maps between this pseudo name to physical  
> > > > > > > > > > name, and  
> > > > > > > > > > the  
> > > > > > > > > > size. The new design allows for more resiliency when it  
> > > > > > > > > > comes  
> > > > > > > > > > to  
> > > > > > > > > > corruption.  
> > > > > > > > > > It also allows for exciting future features like saving a  
> > > > > > > > > > commit  
> > > > > > > > > > point  
> > > > > > > > > > and  
> > > > > > > > > > restoring from it, or automatically create a commit point  
> > > > > > > > > > each  
> > > > > > > > > > day  
> > > > > > > > > > for  
> > > > > > > > > > the  
> > > > > > > > > > last 5 days and be able to rollback to a specific commit  
> > > > > > > > > > point.
> 
> > > > > > > > > > The aim is to create a gateway storage that is going to  
> > > > > > > > > > be  
> > > > > > > > > > the  
> > > > > > > > > > final  
> > > > > > > > > > version, and resilient for future changes. It takes some  
> > > > > > > > > > time  
> > > > > > > > > > to get  
> > > > > > > > > > there,  
> > > > > > > > > > but once we are there, I can safely stand behind using  
> > > > > > > > > > elasticsearch  
> > > > > > > > > > as  
> > > > > > > > > > the  
> > > > > > > > > > main storage as well as releasing v1.0 (I think  
> > > > > > > > > > elasticsearch  
> > > > > > > > > > has  
> > > > > > > > > > enough  
> > > > > > > > > > features for 1.0, just the stability of the gateway is  
> > > > > > > > > > needed).
> 
> > > > > > > > > > I would love for people to take this for a ride and  
> > > > > > > > > > check it  
> > > > > > > > > > out.  
> > > > > > > > > > The  
> > > > > > > > > > next version, as a result of that is going to be 0.10, and  
> > > > > > > > > > I  
> > > > > > > > > > will  
> > > > > > > > > > release  
> > > > > > > > > > it  
> > > > > > > > > > in the following days.
> 
> > > > > > > > > > -shay.banon

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 25, 2010, 6:55pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245/13 "2010-08-25T18:55:05Z")

</div>

Yes, here it is: [upgrade elasticsearch from 0.9.x to 0.10.0 script · GitHub](http://gist.github.com/546494).

-shay.banon

On Wed, Aug 25, 2010 at 8:46 PM, Paul [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote:

> Hi Shay,  
> Just curious, is the conversion script for file based gateway is  
> available? I haven't searched through GIT, so maybe it is there?
> 
> Spent the weekend building up 20mil docs and would prefer not to  
> repeat this.
> 
> Thanks,  
> Paul
> 
> On Aug 24, 4:34 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > Indeed. All the \_\_xxx fils are binary files (either index files or  
> > transaction log parts). The commit-N is a json file that provides meta  
> > information on the commit point.
> > 
> > On Wed, Aug 25, 2010 at 1:32 AM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com) wrote:
> > 
> > > I was very curious to see what the file format was!
> > 
> > > On Aug 24, 3:05 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > > 
> > > > Great, thanks for validating that. Did not expect someone to open vi  
> > > > on  
> > > > the  
> > > > gateway 😉
> > 
> > > > On Wed, Aug 25, 2010 at 12:18 AM, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com)  
> > > > wrote:
> > > > 
> > > > > I think it's fixed in head; I didn't see this error again when  
> > > > > trying  
> > > > > the test below.
> > 
> > > > > On Aug 24, 1:12 pm, Grant Rodgers [gra...@gmail.com](mailto:gra...@gmail.com) wrote:
> > > > > 
> > > > > > Oh you know it was probably a vi swap file. I was taking a look  
> > > > > > at  
> > > > > > one  
> > > > > > of the commit logs, and it might have tried to snapshot while it  
> > > > > > was  
> > > > > > open.
> > 
> > > > > > I think you have committed another change since 80c7135 that  
> > > > > > ignores  
> > > > > > files elasticsearch didn't create. I'll build the latest head and  
> > > > > > try  
> > > > > > viewing a commit snapshot again.
> > 
> > > > > > On Aug 24, 1:00 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)  
> > > > > > wrote:
> > 
> > > > > > > Strange, not sure how this file ended up in the gateway, I  
> > > > > > > can't  
> > > > > > > see  
> > > > > > > where  
> > > > > > > elasticsearch would write it. It only writes \_\_xxx files (no .  
> > > > > > > something)  
> > > > > > > and commit- files. I will fix it to ignore files that don't  
> > > > > > > conform  
> > > > > > > to  
> > > > > > > the  
> > > > > > > format, but we should try and understand where its coming  
> > > > > > > from...
> > 
> > > > > > > -shay.banon
> > 
> > > > > > > On Tue, Aug 24, 2010 at 10:25 PM, Grant Rodgers \<  
> > > > > > > [gra...@gmail.com](mailto:gra...@gmail.com)\>  
> > > > > > > wrote:
> > > > > > > 
> > > > > > > > Oh btw that error was with 80c7135
> > 
> > > > > > > > On Aug 23, 12:22 pm, Shay Banon \<  
> > > > > > > > [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)\>  
> > > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > Yes, though there is a whole "workflow" level APIs and  
> > > > > > > > > support  
> > > > > > > > > for.  
> > > > > > > > > But,  
> > > > > > > > > the  
> > > > > > > > > basics are there in the gateway.
> > 
> > > > > > > > > -shay.banon
> > 
> > > > > > > > > On Mon, Aug 23, 2010 at 10:21 PM, Tal \<  
> > > > > > > > > [talsalm...@gmail.com](mailto:talsalm...@gmail.com)\>  
> > > > > > > > > wrote:
> > > > > > > > > 
> > > > > > > > > > Very nice feature indeed.  
> > > > > > > > > > Will this also allow for index level commit points?
> > 
> > > > > > > > > > Tal
> > 
> > > > > > > > > > On Aug 23, 10:11 pm, Shay Banon \<  
> > > > > > > > > > [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)\>  
> > > > > > > > > > wrote:
> > > > > > > > > > 
> > > > > > > > > > > Hi,
> > 
> > > > > > > > > > > ```
> > > > > > > > > > > I am going to push (pretty soon) a major rewrite of
> > > > > > > > > > > 
> > > > > > > > > > > ```
> 
> the
> 
> > > > > gateway
> > > > > 
> > > > > > > > > > module
> > > > > > > > > > 
> > > > > > > > > > > and improved throttling support. The gateway change is  
> > > > > > > > > > > a  
> > > > > > > > > > > breaking  
> > > > > > > > > > > change,  
> > > > > > > > > > > meaning that new version will not be able to recover  
> > > > > > > > > > > from a  
> > > > > > > > > > > 0.9  
> > > > > > > > > > > gateway.  
> > > > > > > > > > > I  
> > > > > > > > > > > will provide an upgrade script for file based gateway,  
> > > > > > > > > > > s3  
> > > > > > > > > > > based  
> > > > > > > > > > > gateway  
> > > > > > > > > > > will  
> > > > > > > > > > > require reindexing, though, potentially, that script  
> > > > > > > > > > > can be  
> > > > > > > > > > > adjusted  
> > > > > > > > > > > to  
> > > > > > > > > > > support it.
> > 
> > > > > > > > > > > Let me explain some of the changes. The first is  
> > > > > > > > > > > throttling  
> > > > > > > > > > > support.  
> > > > > > > > > > > In  
> > > > > > > > > > > 0.9, recoveries are being throttled on a specific node  
> > > > > > > > > > > in  
> > > > > > > > > > > order  
> > > > > > > > > > > to  
> > > > > > > > > > > reduce  
> > > > > > > > > > > the load a that node. The throttling was done on the  
> > > > > > > > > > > node  
> > > > > > > > > > > level,  
> > > > > > > > > > > after a  
> > > > > > > > > > > shard has been allocated to it. Maintain the count of  
> > > > > > > > > > > current  
> > > > > > > > > > > recoveries  
> > > > > > > > > > > is  
> > > > > > > > > > > quite tricky because of the complexity of the recovery  
> > > > > > > > > > > process.  
> > > > > > > > > > > This  
> > > > > > > > > > > has  
> > > > > > > > > > > now  
> > > > > > > > > > > been refactored into a better place, which is the  
> > > > > > > > > > > actual  
> > > > > > > > > > > allocation algorithm that runs and shuffles shards  
> > > > > > > > > > > around.
> > 
> > > > > > > > > > > The more interesting change is the gateway. There  
> > > > > > > > > > > were  
> > > > > > > > > > > several  
> > > > > > > > > > > problems  
> > > > > > > > > > > with how the gateway works today that were exposed by a  
> > > > > > > > > > > user of  
> > > > > > > > > > > elasticsearch that stores 4TB data (several indices,  
> > > > > > > > > > > each  
> > > > > > > > > > > with  
> > > > > > > > > > > 10  
> > > > > > > > > > > shards  
> > > > > > > > > > > and  
> > > > > > > > > > > 2 replicas, which sums it up to 12TB). This has  
> > > > > > > > > > > uncovered  
> > > > > > > > > > > some  
> > > > > > > > > > > problems  
> > > > > > > > > > > with  
> > > > > > > > > > > the current design, specifically how md5 are computed  
> > > > > > > > > > > (and  
> > > > > > > > > > > the  
> > > > > > > > > > > time  
> > > > > > > > > > > it  
> > > > > > > > > > > takes  
> > > > > > > > > > > to compute them on the local storage on ec2), as well  
> > > > > > > > > > > as  
> > > > > > > > > > > other  
> > > > > > > > > > > possibilities  
> > > > > > > > > > > for gateway corruptions using this load. Of course,  
> > > > > > > > > > > elasticsearch aim  
> > > > > > > > > > > is  
> > > > > > > > > > > to  
> > > > > > > > > > > be able to store much more data than that, we are  
> > > > > > > > > > > getting  
> > > > > > > > > > > there... .
> > 
> > > > > > > > > > > In general, the new implementation works (in spirit)  
> > > > > > > > > > > in  
> > > > > > > > > > > the  
> > > > > > > > > > > same  
> > > > > > > > > > > manner  
> > > > > > > > > > > git works. Each snapshot is a commit point, that stores  
> > > > > > > > > > > files  
> > > > > > > > > > > in the  
> > > > > > > > > > > gateway  
> > > > > > > > > > > into an auto generated name, and finally, a commit  
> > > > > > > > > > > point is  
> > > > > > > > > > > written  
> > > > > > > > > > > with  
> > > > > > > > > > > the  
> > > > > > > > > > > "directory" which maps between this pseudo name to  
> > > > > > > > > > > physical  
> > > > > > > > > > > name, and  
> > > > > > > > > > > the  
> > > > > > > > > > > size. The new design allows for more resiliency when it  
> > > > > > > > > > > comes  
> > > > > > > > > > > to  
> > > > > > > > > > > corruption.  
> > > > > > > > > > > It also allows for exciting future features like saving  
> > > > > > > > > > > a  
> > > > > > > > > > > commit  
> > > > > > > > > > > point  
> > > > > > > > > > > and  
> > > > > > > > > > > restoring from it, or automatically create a commit  
> > > > > > > > > > > point  
> > > > > > > > > > > each  
> > > > > > > > > > > day  
> > > > > > > > > > > for  
> > > > > > > > > > > the  
> > > > > > > > > > > last 5 days and be able to rollback to a specific  
> > > > > > > > > > > commit  
> > > > > > > > > > > point.
> > 
> > > > > > > > > > > The aim is to create a gateway storage that is going  
> > > > > > > > > > > to  
> > > > > > > > > > > be  
> > > > > > > > > > > the  
> > > > > > > > > > > final  
> > > > > > > > > > > version, and resilient for future changes. It takes  
> > > > > > > > > > > some  
> > > > > > > > > > > time  
> > > > > > > > > > > to get  
> > > > > > > > > > > there,  
> > > > > > > > > > > but once we are there, I can safely stand behind using  
> > > > > > > > > > > elasticsearch  
> > > > > > > > > > > as  
> > > > > > > > > > > the  
> > > > > > > > > > > main storage as well as releasing v1.0 (I think  
> > > > > > > > > > > elasticsearch  
> > > > > > > > > > > has  
> > > > > > > > > > > enough  
> > > > > > > > > > > features for 1.0, just the stability of the gateway is  
> > > > > > > > > > > needed).
> > 
> > > > > > > > > > > I would love for people to take this for a ride and  
> > > > > > > > > > > check it  
> > > > > > > > > > > out.  
> > > > > > > > > > > The  
> > > > > > > > > > > next version, as a result of that is going to be 0.10,  
> > > > > > > > > > > and  
> > > > > > > > > > > I  
> > > > > > > > > > > will  
> > > > > > > > > > > release  
> > > > > > > > > > > it  
> > > > > > > > > > > in the following days.
> > 
> > > > > > > > > > > -shay.banon

---

<div class="post-metadata">

**Author:** ![Kenneth\_Loafman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman/32/2224_2.png) [@Kenneth\_Loafman](https://discuss.elastic.co/u/Kenneth_Loafman)\
**Post date:** [August 25, 2010, 11:22pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245/14 "2010-08-25T23:22:52Z")

</div>

Any hints for how to upgrade an S3 gateway?

...Ken

Shay Banon wrote:

> Yes, here it is: [upgrade elasticsearch from 0.9.x to 0.10.0 script · GitHub](http://gist.github.com/546494).
> 
> -shay.banon
> 
> On Wed, Aug 25, 2010 at 8:46 PM, Paul \<[ppearcy@gmail.com](mailto:ppearcy@gmail.com)  
> [mailto:ppearcy@gmail.com](mailto:ppearcy@gmail.com)\> wrote:
> 
> ```
> Hi Shay,
> Just curious, is the conversion script for file based gateway is
> available? I haven't searched through GIT, so maybe it is there?
> 
> Spent the weekend building up 20mil docs and would prefer not to
> repeat this.
> 
> Thanks,
> Paul
> 
> On Aug 24, 4:34 pm, Shay Banon <shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>> wrote:
> > Indeed. All the __xxx fils are binary files (either index files or
> > transaction log parts). The commit-N is a json file that provides meta
> > information on the commit point.
> >
> >
> >
> > On Wed, Aug 25, 2010 at 1:32 AM, Grant Rodgers <gra...@gmail.com
> <mailto:gra...@gmail.com>> wrote:
> > > I was very curious to see what the file format was!
> >
> > > On Aug 24, 3:05 pm, Shay Banon <shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>> wrote:
> > > > Great, thanks for validating that. Did not expect someone to
> open vi on
> > > the
> > > > gateway ;)
> >
> > > > On Wed, Aug 25, 2010 at 12:18 AM, Grant Rodgers
> <gra...@gmail.com <mailto:gra...@gmail.com>>
> > > wrote:
> > > > > I think it's fixed in head; I didn't see this error again
> when trying
> > > > > the test below.
> >
> > > > > On Aug 24, 1:12 pm, Grant Rodgers <gra...@gmail.com
> <mailto:gra...@gmail.com>> wrote:
> > > > > > Oh you know it was probably a vi swap file. I was taking a
> look at
> > > one
> > > > > > of the commit logs, and it might have tried to snapshot
> while it was
> > > > > > open.
> >
> > > > > > I think you have committed another change since 80c7135
> that ignores
> > > > > > files elasticsearch didn't create. I'll build the latest
> head and try
> > > > > > viewing a commit snapshot again.
> >
> > > > > > On Aug 24, 1:00 pm, Shay Banon
> <shay.ba...@elasticsearch.com <mailto:shay.ba...@elasticsearch.com>>
> wrote:
> >
> > > > > > > Strange, not sure how this file ended up in the gateway,
> I can't
> > > see
> > > > > where
> > > > > > > elasticsearch would write it. It only writes __xxx files
> (no .
> > > > > something)
> > > > > > > and commit- files. I will fix it to ignore files that
> don't conform
> > > to
> > > > > the
> > > > > > > format, but we should try and understand where its
> coming from...
> >
> > > > > > > -shay.banon
> >
> > > > > > > On Tue, Aug 24, 2010 at 10:25 PM, Grant Rodgers
> <gra...@gmail.com <mailto:gra...@gmail.com>>
> > > > > wrote:
> > > > > > > > Oh btw that error was with 80c7135
> >
> > > > > > > > On Aug 23, 12:22 pm, Shay Banon
> <shay.ba...@elasticsearch.com <mailto:shay.ba...@elasticsearch.com>>
> > > > > wrote:
> > > > > > > > > Yes, though there is a whole "workflow" level APIs
> and support
> > > for.
> > > > > But,
> > > > > > > > the
> > > > > > > > > basics are there in the gateway.
> >
> > > > > > > > > -shay.banon
> >
> > > > > > > > > On Mon, Aug 23, 2010 at 10:21 PM, Tal
> <talsalm...@gmail.com <mailto:talsalm...@gmail.com>>
> > > > > wrote:
> > > > > > > > > > Very nice feature indeed.
> > > > > > > > > > Will this also allow for index level commit points?
> >
> > > > > > > > > > Tal
> >
> > > > > > > > > > On Aug 23, 10:11 pm, Shay Banon <
> > > shay.ba...@elasticsearch.com <mailto:shay.ba...@elasticsearch.com>>
> > > > > wrote:
> > > > > > > > > > > Hi,
> >
> > > > > > > > > > > I am going to push (pretty soon) a major
> rewrite of the
> > > > > gateway
> > > > > > > > > > module
> > > > > > > > > > > and improved throttling support. The gateway
> change is a
> > > > > breaking
> > > > > > > > change,
> > > > > > > > > > > meaning that new version will not be able to
> recover from a
> > > 0.9
> > > > > > > > gateway.
> > > > > > > > > > I
> > > > > > > > > > > will provide an upgrade script for file based
> gateway, s3
> > > based
> > > > > > > > gateway
> > > > > > > > > > will
> > > > > > > > > > > require reindexing, though, potentially, that
> script can be
> > > > > adjusted
> > > > > > > > to
> > > > > > > > > > > support it.
> >
> > > > > > > > > > > Let me explain some of the changes. The first is
> > > throttling
> > > > > > > > support.
> > > > > > > > > > In
> > > > > > > > > > > 0.9, recoveries are being throttled on a
> specific node in
> > > order
> > > > > to
> > > > > > > > reduce
> > > > > > > > > > > the load a that node. The throttling was done on
> the node
> > > > > level,
> > > > > > > > after a
> > > > > > > > > > > shard has been allocated to it. Maintain the
> count of
> > > current
> > > > > > > > recoveries
> > > > > > > > > > is
> > > > > > > > > > > quite tricky because of the complexity of the
> recovery
> > > process.
> > > > > This
> > > > > > > > has
> > > > > > > > > > now
> > > > > > > > > > > been refactored into a better place, which is
> the actual
> > > > > > > > > > > allocation algorithm that runs and shuffles
> shards around.
> >
> > > > > > > > > > > The more interesting change is the gateway.
> There were
> > > > > several
> > > > > > > > > > problems
> > > > > > > > > > > with how the gateway works today that were
> exposed by a
> > > user of
> > > > > > > > > > > elasticsearch that stores 4TB data (several
> indices, each
> > > with
> > > > > 10
> > > > > > > > shards
> > > > > > > > > > and
> > > > > > > > > > > 2 replicas, which sums it up to 12TB). This has
> uncovered
> > > some
> > > > > > > > problems
> > > > > > > > > > with
> > > > > > > > > > > the current design, specifically how md5 are
> computed (and
> > > the
> > > > > time
> > > > > > > > it
> > > > > > > > > > takes
> > > > > > > > > > > to compute them on the local storage on ec2), as
> well as
> > > other
> > > > > > > > > > possibilities
> > > > > > > > > > > for gateway corruptions using this load. Of course,
> > > > > elasticsearch aim
> > > > > > > > is
> > > > > > > > > > to
> > > > > > > > > > > be able to store much more data than that, we
> are getting
> > > > > there... .
> >
> > > > > > > > > > > In general, the new implementation works (in
> spirit) in
> > > the
> > > > > same
> > > > > > > > > > manner
> > > > > > > > > > > git works. Each snapshot is a commit point, that
> stores
> > > files
> > > > > in the
> > > > > > > > > > gateway
> > > > > > > > > > > into an auto generated name, and finally, a
> commit point is
> > > > > written
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > "directory" which maps between this pseudo name
> to physical
> > > > > name, and
> > > > > > > > the
> > > > > > > > > > > size. The new design allows for more resiliency
> when it
> > > comes
> > > > > to
> > > > > > > > > > corruption.
> > > > > > > > > > > It also allows for exciting future features like
> saving a
> > > > > commit
> > > > > > > > point
> > > > > > > > > > and
> > > > > > > > > > > restoring from it, or automatically create a
> commit point
> > > each
> > > > > day
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > last 5 days and be able to rollback to a
> specific commit
> > > point.
> >
> > > > > > > > > > > The aim is to create a gateway storage that
> is going to
> > > be
> > > > > the
> > > > > > > > final
> > > > > > > > > > > version, and resilient for future changes. It
> takes some
> > > time
> > > > > to get
> > > > > > > > > > there,
> > > > > > > > > > > but once we are there, I can safely stand behind
> using
> > > > > elasticsearch
> > > > > > > > as
> > > > > > > > > > the
> > > > > > > > > > > main storage as well as releasing v1.0 (I think
> > > elasticsearch
> > > > > has
> > > > > > > > enough
> > > > > > > > > > > features for 1.0, just the stability of the
> gateway is
> > > needed).
> >
> > > > > > > > > > > I would love for people to take this for a
> ride and
> > > check it
> > > > > out.
> > > > > > > > The
> > > > > > > > > > > next version, as a result of that is going to be
> 0.10, and
> > > I
> > > > > will
> > > > > > > > release
> > > > > > > > > > it
> > > > > > > > > > > in the following days.
> >
> > > > > > > > > > > -shay.banon
> 
> ```

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 25, 2010, 11:26pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245/15 "2010-08-25T23:26:33Z")

</div>

You can check what I do in the script, its pretty simple. The problem is  
that S3 does not have a rename method..., though maybe a copy command can be  
used, but not sure how long it takes for large files. You can replace the  
file based operations with S3 ones.

-shay.banon

On Thu, Aug 26, 2010 at 2:22 AM, Kenneth Loafman  
[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)wrote:

> Any hints for how to upgrade an S3 gateway?
> 
> ...Ken
> 
> Shay Banon wrote:
> 
> > Yes, here it is: [upgrade elasticsearch from 0.9.x to 0.10.0 script · GitHub](http://gist.github.com/546494).
> > 
> > -shay.banon
> > 
> > On Wed, Aug 25, 2010 at 8:46 PM, Paul \<[ppearcy@gmail.com](mailto:ppearcy@gmail.com)  
> > [mailto:ppearcy@gmail.com](mailto:ppearcy@gmail.com)\> wrote:
> > 
> > ```
> > Hi Shay,
> > Just curious, is the conversion script for file based gateway is
> > available? I haven't searched through GIT, so maybe it is there?
> > 
> > Spent the weekend building up 20mil docs and would prefer not to
> > repeat this.
> > 
> > Thanks,
> > Paul
> > 
> > On Aug 24, 4:34 pm, Shay Banon <shay.ba...@elasticsearch.com
> > <mailto:shay.ba...@elasticsearch.com>> wrote:
> > > Indeed. All the __xxx fils are binary files (either index files or
> > > transaction log parts). The commit-N is a json file that provides
> > 
> > ```
> 
> meta
> 
> > ```
> > > information on the commit point.
> > >
> > >
> > >
> > > On Wed, Aug 25, 2010 at 1:32 AM, Grant Rodgers <gra...@gmail.com
> > <mailto:gra...@gmail.com>> wrote:
> > > > I was very curious to see what the file format was!
> > >
> > > > On Aug 24, 3:05 pm, Shay Banon <shay.ba...@elasticsearch.com
> > <mailto:shay.ba...@elasticsearch.com>> wrote:
> > > > > Great, thanks for validating that. Did not expect someone to
> > open vi on
> > > > the
> > > > > gateway ;)
> > >
> > > > > On Wed, Aug 25, 2010 at 12:18 AM, Grant Rodgers
> > <gra...@gmail.com <mailto:gra...@gmail.com>>
> > > > wrote:
> > > > > > I think it's fixed in head; I didn't see this error again
> > when trying
> > > > > > the test below.
> > >
> > > > > > On Aug 24, 1:12 pm, Grant Rodgers <gra...@gmail.com
> > <mailto:gra...@gmail.com>> wrote:
> > > > > > > Oh you know it was probably a vi swap file. I was taking a
> > look at
> > > > one
> > > > > > > of the commit logs, and it might have tried to snapshot
> > while it was
> > > > > > > open.
> > >
> > > > > > > I think you have committed another change since 80c7135
> > that ignores
> > > > > > > files elasticsearch didn't create. I'll build the latest
> > head and try
> > > > > > > viewing a commit snapshot again.
> > >
> > > > > > > On Aug 24, 1:00 pm, Shay Banon
> > <shay.ba...@elasticsearch.com <mailto:shay.ba...@elasticsearch.com>>
> > wrote:
> > >
> > > > > > > > Strange, not sure how this file ended up in the gateway,
> > I can't
> > > > see
> > > > > > where
> > > > > > > > elasticsearch would write it. It only writes __xxx files
> > (no .
> > > > > > something)
> > > > > > > > and commit- files. I will fix it to ignore files that
> > don't conform
> > > > to
> > > > > > the
> > > > > > > > format, but we should try and understand where its
> > coming from...
> > >
> > > > > > > > -shay.banon
> > >
> > > > > > > > On Tue, Aug 24, 2010 at 10:25 PM, Grant Rodgers
> > <gra...@gmail.com <mailto:gra...@gmail.com>>
> > > > > > wrote:
> > > > > > > > > Oh btw that error was with 80c7135
> > >
> > > > > > > > > On Aug 23, 12:22 pm, Shay Banon
> > <shay.ba...@elasticsearch.com <mailto:shay.ba...@elasticsearch.com>>
> > > > > > wrote:
> > > > > > > > > > Yes, though there is a whole "workflow" level APIs
> > and support
> > > > for.
> > > > > > But,
> > > > > > > > > the
> > > > > > > > > > basics are there in the gateway.
> > >
> > > > > > > > > > -shay.banon
> > >
> > > > > > > > > > On Mon, Aug 23, 2010 at 10:21 PM, Tal
> > <talsalm...@gmail.com <mailto:talsalm...@gmail.com>>
> > > > > > wrote:
> > > > > > > > > > > Very nice feature indeed.
> > > > > > > > > > > Will this also allow for index level commit points?
> > >
> > > > > > > > > > > Tal
> > >
> > > > > > > > > > > On Aug 23, 10:11 pm, Shay Banon <
> > > > shay.ba...@elasticsearch.com <mailto:
> > 
> > ```
> 
> [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)\>\>
> 
> > ```
> > > > > > wrote:
> > > > > > > > > > > > Hi,
> > >
> > > > > > > > > > > > I am going to push (pretty soon) a major
> > rewrite of the
> > > > > > gateway
> > > > > > > > > > > module
> > > > > > > > > > > > and improved throttling support. The gateway
> > change is a
> > > > > > breaking
> > > > > > > > > change,
> > > > > > > > > > > > meaning that new version will not be able to
> > recover from a
> > > > 0.9
> > > > > > > > > gateway.
> > > > > > > > > > > I
> > > > > > > > > > > > will provide an upgrade script for file based
> > gateway, s3
> > > > based
> > > > > > > > > gateway
> > > > > > > > > > > will
> > > > > > > > > > > > require reindexing, though, potentially, that
> > script can be
> > > > > > adjusted
> > > > > > > > > to
> > > > > > > > > > > > support it.
> > >
> > > > > > > > > > > > Let me explain some of the changes. The first
> > 
> > ```
> 
> is
> 
> > ```
> > > > throttling
> > > > > > > > > support.
> > > > > > > > > > > In
> > > > > > > > > > > > 0.9, recoveries are being throttled on a
> > specific node in
> > > > order
> > > > > > to
> > > > > > > > > reduce
> > > > > > > > > > > > the load a that node. The throttling was done on
> > the node
> > > > > > level,
> > > > > > > > > after a
> > > > > > > > > > > > shard has been allocated to it. Maintain the
> > count of
> > > > current
> > > > > > > > > recoveries
> > > > > > > > > > > is
> > > > > > > > > > > > quite tricky because of the complexity of the
> > recovery
> > > > process.
> > > > > > This
> > > > > > > > > has
> > > > > > > > > > > now
> > > > > > > > > > > > been refactored into a better place, which is
> > the actual
> > > > > > > > > > > > allocation algorithm that runs and shuffles
> > shards around.
> > >
> > > > > > > > > > > > The more interesting change is the gateway.
> > There were
> > > > > > several
> > > > > > > > > > > problems
> > > > > > > > > > > > with how the gateway works today that were
> > exposed by a
> > > > user of
> > > > > > > > > > > > elasticsearch that stores 4TB data (several
> > indices, each
> > > > with
> > > > > > 10
> > > > > > > > > shards
> > > > > > > > > > > and
> > > > > > > > > > > > 2 replicas, which sums it up to 12TB). This has
> > uncovered
> > > > some
> > > > > > > > > problems
> > > > > > > > > > > with
> > > > > > > > > > > > the current design, specifically how md5 are
> > computed (and
> > > > the
> > > > > > time
> > > > > > > > > it
> > > > > > > > > > > takes
> > > > > > > > > > > > to compute them on the local storage on ec2), as
> > well as
> > > > other
> > > > > > > > > > > possibilities
> > > > > > > > > > > > for gateway corruptions using this load. Of
> > 
> > ```
> 
> course,
> 
> > ```
> > > > > > elasticsearch aim
> > > > > > > > > is
> > > > > > > > > > > to
> > > > > > > > > > > > be able to store much more data than that, we
> > are getting
> > > > > > there... .
> > >
> > > > > > > > > > > > In general, the new implementation works (in
> > spirit) in
> > > > the
> > > > > > same
> > > > > > > > > > > manner
> > > > > > > > > > > > git works. Each snapshot is a commit point, that
> > stores
> > > > files
> > > > > > in the
> > > > > > > > > > > gateway
> > > > > > > > > > > > into an auto generated name, and finally, a
> > commit point is
> > > > > > written
> > > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > > "directory" which maps between this pseudo name
> > to physical
> > > > > > name, and
> > > > > > > > > the
> > > > > > > > > > > > size. The new design allows for more resiliency
> > when it
> > > > comes
> > > > > > to
> > > > > > > > > > > corruption.
> > > > > > > > > > > > It also allows for exciting future features like
> > saving a
> > > > > > commit
> > > > > > > > > point
> > > > > > > > > > > and
> > > > > > > > > > > > restoring from it, or automatically create a
> > commit point
> > > > each
> > > > > > day
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > last 5 days and be able to rollback to a
> > specific commit
> > > > point.
> > >
> > > > > > > > > > > > The aim is to create a gateway storage that
> > is going to
> > > > be
> > > > > > the
> > > > > > > > > final
> > > > > > > > > > > > version, and resilient for future changes. It
> > takes some
> > > > time
> > > > > > to get
> > > > > > > > > > > there,
> > > > > > > > > > > > but once we are there, I can safely stand behind
> > using
> > > > > > elasticsearch
> > > > > > > > > as
> > > > > > > > > > > the
> > > > > > > > > > > > main storage as well as releasing v1.0 (I think
> > > > elasticsearch
> > > > > > has
> > > > > > > > > enough
> > > > > > > > > > > > features for 1.0, just the stability of the
> > gateway is
> > > > needed).
> > >
> > > > > > > > > > > > I would love for people to take this for a
> > ride and
> > > > check it
> > > > > > out.
> > > > > > > > > The
> > > > > > > > > > > > next version, as a result of that is going to be
> > 0.10, and
> > > > I
> > > > > > will
> > > > > > > > > release
> > > > > > > > > > > it
> > > > > > > > > > > > in the following days.
> > >
> > > > > > > > > > > > -shay.banon
> > 
> > ```

---

<div class="post-metadata">

**Author:** ![Kenneth\_Loafman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman/32/2224_2.png) [@Kenneth\_Loafman](https://discuss.elastic.co/u/Kenneth_Loafman)\
**Post date:** [August 26, 2010, 12:30pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245/16 "2010-08-26T12:30:47Z")

</div>

Is it possible to copy a gateway from S3 to local, run the conversion  
locally, then copy it back to S3? The S3 rename issue could be solved  
this way, with less total downtime.

...Ken

Shay Banon wrote:

> You can check what I do in the script, its pretty simple. The problem is  
> that S3 does not have a rename method..., though maybe a copy command  
> can be used, but not sure how long it takes for large files. You can  
> replace the file based operations with S3 ones.
> 
> -shay.banon
> 
> On Thu, Aug 26, 2010 at 2:22 AM, Kenneth Loafman  
> \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> 
> ```
> Any hints for how to upgrade an S3 gateway?
> 
> ...Ken
> 
> Shay Banon wrote:
> > Yes, here it is: http://gist.github.com/546494.
> >
> > -shay.banon
> >
> > On Wed, Aug 25, 2010 at 8:46 PM, Paul <ppearcy@gmail.com
> <mailto:ppearcy@gmail.com>
> > <mailto:ppearcy@gmail.com <mailto:ppearcy@gmail.com>>> wrote:
> >
> > Hi Shay,
> > Just curious, is the conversion script for file based gateway is
> > available? I haven't searched through GIT, so maybe it is there?
> >
> > Spent the weekend building up 20mil docs and would prefer not to
> > repeat this.
> >
> > Thanks,
> > Paul
> >
> > On Aug 24, 4:34 pm, Shay Banon <shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>
> > <mailto:shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>>> wrote:
> > > Indeed. All the __xxx fils are binary files (either index
> files or
> > > transaction log parts). The commit-N is a json file that
> provides meta
> > > information on the commit point.
> > >
> > >
> > >
> > > On Wed, Aug 25, 2010 at 1:32 AM, Grant Rodgers
> <gra...@gmail.com <mailto:gra...@gmail.com>
> > <mailto:gra...@gmail.com <mailto:gra...@gmail.com>>> wrote:
> > > > I was very curious to see what the file format was!
> > >
> > > > On Aug 24, 3:05 pm, Shay Banon
> <shay.ba...@elasticsearch.com <mailto:shay.ba...@elasticsearch.com>
> > <mailto:shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>>> wrote:
> > > > > Great, thanks for validating that. Did not expect someone to
> > open vi on
> > > > the
> > > > > gateway ;)
> > >
> > > > > On Wed, Aug 25, 2010 at 12:18 AM, Grant Rodgers
> > <gra...@gmail.com <mailto:gra...@gmail.com>
> <mailto:gra...@gmail.com <mailto:gra...@gmail.com>>>
> > > > wrote:
> > > > > > I think it's fixed in head; I didn't see this error again
> > when trying
> > > > > > the test below.
> > >
> > > > > > On Aug 24, 1:12 pm, Grant Rodgers <gra...@gmail.com
> <mailto:gra...@gmail.com>
> > <mailto:gra...@gmail.com <mailto:gra...@gmail.com>>> wrote:
> > > > > > > Oh you know it was probably a vi swap file. I was
> taking a
> > look at
> > > > one
> > > > > > > of the commit logs, and it might have tried to snapshot
> > while it was
> > > > > > > open.
> > >
> > > > > > > I think you have committed another change since 80c7135
> > that ignores
> > > > > > > files elasticsearch didn't create. I'll build the latest
> > head and try
> > > > > > > viewing a commit snapshot again.
> > >
> > > > > > > On Aug 24, 1:00 pm, Shay Banon
> > <shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>
> <mailto:shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>>>
> > wrote:
> > >
> > > > > > > > Strange, not sure how this file ended up in the
> gateway,
> > I can't
> > > > see
> > > > > > where
> > > > > > > > elasticsearch would write it. It only writes __xxx
> files
> > (no .
> > > > > > something)
> > > > > > > > and commit- files. I will fix it to ignore files that
> > don't conform
> > > > to
> > > > > > the
> > > > > > > > format, but we should try and understand where its
> > coming from...
> > >
> > > > > > > > -shay.banon
> > >
> > > > > > > > On Tue, Aug 24, 2010 at 10:25 PM, Grant Rodgers
> > <gra...@gmail.com <mailto:gra...@gmail.com>
> <mailto:gra...@gmail.com <mailto:gra...@gmail.com>>>
> > > > > > wrote:
> > > > > > > > > Oh btw that error was with 80c7135
> > >
> > > > > > > > > On Aug 23, 12:22 pm, Shay Banon
> > <shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>
> <mailto:shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>>>
> > > > > > wrote:
> > > > > > > > > > Yes, though there is a whole "workflow" level APIs
> > and support
> > > > for.
> > > > > > But,
> > > > > > > > > the
> > > > > > > > > > basics are there in the gateway.
> > >
> > > > > > > > > > -shay.banon
> > >
> > > > > > > > > > On Mon, Aug 23, 2010 at 10:21 PM, Tal
> > <talsalm...@gmail.com <mailto:talsalm...@gmail.com>
> <mailto:talsalm...@gmail.com <mailto:talsalm...@gmail.com>>>
> > > > > > wrote:
> > > > > > > > > > > Very nice feature indeed.
> > > > > > > > > > > Will this also allow for index level commit
> points?
> > >
> > > > > > > > > > > Tal
> > >
> > > > > > > > > > > On Aug 23, 10:11 pm, Shay Banon <
> > > > shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>
> <mailto:shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>>>
> > > > > > wrote:
> > > > > > > > > > > > Hi,
> > >
> > > > > > > > > > > > I am going to push (pretty soon) a major
> > rewrite of the
> > > > > > gateway
> > > > > > > > > > > module
> > > > > > > > > > > > and improved throttling support. The gateway
> > change is a
> > > > > > breaking
> > > > > > > > > change,
> > > > > > > > > > > > meaning that new version will not be able to
> > recover from a
> > > > 0.9
> > > > > > > > > gateway.
> > > > > > > > > > > I
> > > > > > > > > > > > will provide an upgrade script for file based
> > gateway, s3
> > > > based
> > > > > > > > > gateway
> > > > > > > > > > > will
> > > > > > > > > > > > require reindexing, though, potentially, that
> > script can be
> > > > > > adjusted
> > > > > > > > > to
> > > > > > > > > > > > support it.
> > >
> > > > > > > > > > > > Let me explain some of the changes. The
> first is
> > > > throttling
> > > > > > > > > support.
> > > > > > > > > > > In
> > > > > > > > > > > > 0.9, recoveries are being throttled on a
> > specific node in
> > > > order
> > > > > > to
> > > > > > > > > reduce
> > > > > > > > > > > > the load a that node. The throttling was
> done on
> > the node
> > > > > > level,
> > > > > > > > > after a
> > > > > > > > > > > > shard has been allocated to it. Maintain the
> > count of
> > > > current
> > > > > > > > > recoveries
> > > > > > > > > > > is
> > > > > > > > > > > > quite tricky because of the complexity of the
> > recovery
> > > > process.
> > > > > > This
> > > > > > > > > has
> > > > > > > > > > > now
> > > > > > > > > > > > been refactored into a better place, which is
> > the actual
> > > > > > > > > > > > allocation algorithm that runs and shuffles
> > shards around.
> > >
> > > > > > > > > > > > The more interesting change is the gateway.
> > There were
> > > > > > several
> > > > > > > > > > > problems
> > > > > > > > > > > > with how the gateway works today that were
> > exposed by a
> > > > user of
> > > > > > > > > > > > elasticsearch that stores 4TB data (several
> > indices, each
> > > > with
> > > > > > 10
> > > > > > > > > shards
> > > > > > > > > > > and
> > > > > > > > > > > > 2 replicas, which sums it up to 12TB).
> This has
> > uncovered
> > > > some
> > > > > > > > > problems
> > > > > > > > > > > with
> > > > > > > > > > > > the current design, specifically how md5 are
> > computed (and
> > > > the
> > > > > > time
> > > > > > > > > it
> > > > > > > > > > > takes
> > > > > > > > > > > > to compute them on the local storage on
> ec2), as
> > well as
> > > > other
> > > > > > > > > > > possibilities
> > > > > > > > > > > > for gateway corruptions using this load.
> Of course,
> > > > > > elasticsearch aim
> > > > > > > > > is
> > > > > > > > > > > to
> > > > > > > > > > > > be able to store much more data than that, we
> > are getting
> > > > > > there... .
> > >
> > > > > > > > > > > > In general, the new implementation
> works (in
> > spirit) in
> > > > the
> > > > > > same
> > > > > > > > > > > manner
> > > > > > > > > > > > git works. Each snapshot is a commit
> point, that
> > stores
> > > > files
> > > > > > in the
> > > > > > > > > > > gateway
> > > > > > > > > > > > into an auto generated name, and finally, a
> > commit point is
> > > > > > written
> > > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > > "directory" which maps between this pseudo
> name
> > to physical
> > > > > > name, and
> > > > > > > > > the
> > > > > > > > > > > > size. The new design allows for more
> resiliency
> > when it
> > > > comes
> > > > > > to
> > > > > > > > > > > corruption.
> > > > > > > > > > > > It also allows for exciting future
> features like
> > saving a
> > > > > > commit
> > > > > > > > > point
> > > > > > > > > > > and
> > > > > > > > > > > > restoring from it, or automatically create a
> > commit point
> > > > each
> > > > > > day
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > last 5 days and be able to rollback to a
> > specific commit
> > > > point.
> > >
> > > > > > > > > > > > The aim is to create a gateway storage that
> > is going to
> > > > be
> > > > > > the
> > > > > > > > > final
> > > > > > > > > > > > version, and resilient for future changes. It
> > takes some
> > > > time
> > > > > > to get
> > > > > > > > > > > there,
> > > > > > > > > > > > but once we are there, I can safely stand
> behind
> > using
> > > > > > elasticsearch
> > > > > > > > > as
> > > > > > > > > > > the
> > > > > > > > > > > > main storage as well as releasing v1.0 (I
> think
> > > > elasticsearch
> > > > > > has
> > > > > > > > > enough
> > > > > > > > > > > > features for 1.0, just the stability of the
> > gateway is
> > > > needed).
> > >
> > > > > > > > > > > > I would love for people to take this for a
> > ride and
> > > > check it
> > > > > > out.
> > > > > > > > > The
> > > > > > > > > > > > next version, as a result of that is going
> to be
> > 0.10, and
> > > > I
> > > > > > will
> > > > > > > > > release
> > > > > > > > > > > it
> > > > > > > > > > > > in the following days.
> > >
> > > > > > > > > > > > -shay.banon
> >
> >
> 
> ```

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 26, 2010, 1:27pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245/17 "2010-08-26T13:27:56Z")

</div>

Yea, sure.

On Thu, Aug 26, 2010 at 3:30 PM, Kenneth Loafman  
[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)wrote:

> Is it possible to copy a gateway from S3 to local, run the conversion  
> locally, then copy it back to S3? The S3 rename issue could be solved  
> this way, with less total downtime.
> 
> ...Ken
> 
> Shay Banon wrote:
> 
> > You can check what I do in the script, its pretty simple. The problem is  
> > that S3 does not have a rename method..., though maybe a copy command  
> > can be used, but not sure how long it takes for large files. You can  
> > replace the file based operations with S3 ones.
> > 
> > -shay.banon
> > 
> > On Thu, Aug 26, 2010 at 2:22 AM, Kenneth Loafman  
> > \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> > 
> > ```
> > Any hints for how to upgrade an S3 gateway?
> > 
> > ...Ken
> > 
> > Shay Banon wrote:
> > > Yes, here it is: http://gist.github.com/546494.
> > >
> > > -shay.banon
> > >
> > > On Wed, Aug 25, 2010 at 8:46 PM, Paul <ppearcy@gmail.com
> > <mailto:ppearcy@gmail.com>
> > > <mailto:ppearcy@gmail.com <mailto:ppearcy@gmail.com>>> wrote:
> > >
> > > Hi Shay,
> > > Just curious, is the conversion script for file based gateway
> > 
> > ```
> 
> is
> 
> > ```
> > > available? I haven't searched through GIT, so maybe it is
> > 
> > ```
> 
> there?
> 
> > ```
> > >
> > > Spent the weekend building up 20mil docs and would prefer not
> > 
> > ```
> 
> to
> 
> > ```
> > > repeat this.
> > >
> > > Thanks,
> > > Paul
> > >
> > > On Aug 24, 4:34 pm, Shay Banon <shay.ba...@elasticsearch.com
> > <mailto:shay.ba...@elasticsearch.com>
> > > <mailto:shay.ba...@elasticsearch.com
> > <mailto:shay.ba...@elasticsearch.com>>> wrote:
> > > > Indeed. All the __xxx fils are binary files (either index
> > files or
> > > > transaction log parts). The commit-N is a json file that
> > provides meta
> > > > information on the commit point.
> > > >
> > > >
> > > >
> > > > On Wed, Aug 25, 2010 at 1:32 AM, Grant Rodgers
> > <gra...@gmail.com <mailto:gra...@gmail.com>
> > > <mailto:gra...@gmail.com <mailto:gra...@gmail.com>>> wrote:
> > > > > I was very curious to see what the file format was!
> > > >
> > > > > On Aug 24, 3:05 pm, Shay Banon
> > <shay.ba...@elasticsearch.com <mailto:shay.ba...@elasticsearch.com>
> > > <mailto:shay.ba...@elasticsearch.com
> > <mailto:shay.ba...@elasticsearch.com>>> wrote:
> > > > > > Great, thanks for validating that. Did not expect someone
> > 
> > ```
> 
> to
> 
> > ```
> > > open vi on
> > > > > the
> > > > > > gateway ;)
> > > >
> > > > > > On Wed, Aug 25, 2010 at 12:18 AM, Grant Rodgers
> > > <gra...@gmail.com <mailto:gra...@gmail.com>
> > <mailto:gra...@gmail.com <mailto:gra...@gmail.com>>>
> > > > > wrote:
> > > > > > > I think it's fixed in head; I didn't see this error
> > 
> > ```
> 
> again
> 
> > ```
> > > when trying
> > > > > > > the test below.
> > > >
> > > > > > > On Aug 24, 1:12 pm, Grant Rodgers <gra...@gmail.com
> > <mailto:gra...@gmail.com>
> > > <mailto:gra...@gmail.com <mailto:gra...@gmail.com>>> wrote:
> > > > > > > > Oh you know it was probably a vi swap file. I was
> > taking a
> > > look at
> > > > > one
> > > > > > > > of the commit logs, and it might have tried to
> > 
> > ```
> 
> snapshot
> 
> > ```
> > > while it was
> > > > > > > > open.
> > > >
> > > > > > > > I think you have committed another change since
> > 
> > ```
> 
> 80c7135
> 
> > ```
> > > that ignores
> > > > > > > > files elasticsearch didn't create. I'll build the
> > 
> > ```
> 
> latest
> 
> > ```
> > > head and try
> > > > > > > > viewing a commit snapshot again.
> > > >
> > > > > > > > On Aug 24, 1:00 pm, Shay Banon
> > > <shay.ba...@elasticsearch.com
> > <mailto:shay.ba...@elasticsearch.com>
> > <mailto:shay.ba...@elasticsearch.com
> > <mailto:shay.ba...@elasticsearch.com>>>
> > > wrote:
> > > >
> > > > > > > > > Strange, not sure how this file ended up in the
> > gateway,
> > > I can't
> > > > > see
> > > > > > > where
> > > > > > > > > elasticsearch would write it. It only writes __xxx
> > files
> > > (no .
> > > > > > > something)
> > > > > > > > > and commit- files. I will fix it to ignore files
> > 
> > ```
> 
> that
> 
> > ```
> > > don't conform
> > > > > to
> > > > > > > the
> > > > > > > > > format, but we should try and understand where its
> > > coming from...
> > > >
> > > > > > > > > -shay.banon
> > > >
> > > > > > > > > On Tue, Aug 24, 2010 at 10:25 PM, Grant Rodgers
> > > <gra...@gmail.com <mailto:gra...@gmail.com>
> > <mailto:gra...@gmail.com <mailto:gra...@gmail.com>>>
> > > > > > > wrote:
> > > > > > > > > > Oh btw that error was with 80c7135
> > > >
> > > > > > > > > > On Aug 23, 12:22 pm, Shay Banon
> > > <shay.ba...@elasticsearch.com
> > <mailto:shay.ba...@elasticsearch.com>
> > <mailto:shay.ba...@elasticsearch.com
> > <mailto:shay.ba...@elasticsearch.com>>>
> > > > > > > wrote:
> > > > > > > > > > > Yes, though there is a whole "workflow" level
> > 
> > ```
> 
> APIs
> 
> > ```
> > > and support
> > > > > for.
> > > > > > > But,
> > > > > > > > > > the
> > > > > > > > > > > basics are there in the gateway.
> > > >
> > > > > > > > > > > -shay.banon
> > > >
> > > > > > > > > > > On Mon, Aug 23, 2010 at 10:21 PM, Tal
> > > <talsalm...@gmail.com <mailto:talsalm...@gmail.com>
> > <mailto:talsalm...@gmail.com <mailto:talsalm...@gmail.com>>>
> > > > > > > wrote:
> > > > > > > > > > > > Very nice feature indeed.
> > > > > > > > > > > > Will this also allow for index level commit
> > points?
> > > >
> > > > > > > > > > > > Tal
> > > >
> > > > > > > > > > > > On Aug 23, 10:11 pm, Shay Banon <
> > > > > shay.ba...@elasticsearch.com
> > <mailto:shay.ba...@elasticsearch.com>
> > <mailto:shay.ba...@elasticsearch.com
> > <mailto:shay.ba...@elasticsearch.com>>>
> > > > > > > wrote:
> > > > > > > > > > > > > Hi,
> > > >
> > > > > > > > > > > > > I am going to push (pretty soon) a
> > 
> > ```
> 
> major
> 
> > ```
> > > rewrite of the
> > > > > > > gateway
> > > > > > > > > > > > module
> > > > > > > > > > > > > and improved throttling support. The
> > 
> > ```
> 
> gateway
> 
> > ```
> > > change is a
> > > > > > > breaking
> > > > > > > > > > change,
> > > > > > > > > > > > > meaning that new version will not be able
> > 
> > ```
> 
> to
> 
> > ```
> > > recover from a
> > > > > 0.9
> > > > > > > > > > gateway.
> > > > > > > > > > > > I
> > > > > > > > > > > > > will provide an upgrade script for file
> > 
> > ```
> 
> based
> 
> > ```
> > > gateway, s3
> > > > > based
> > > > > > > > > > gateway
> > > > > > > > > > > > will
> > > > > > > > > > > > > require reindexing, though, potentially,
> > 
> > ```
> 
> that
> 
> > ```
> > > script can be
> > > > > > > adjusted
> > > > > > > > > > to
> > > > > > > > > > > > > support it.
> > > >
> > > > > > > > > > > > > Let me explain some of the changes. The
> > first is
> > > > > throttling
> > > > > > > > > > support.
> > > > > > > > > > > > In
> > > > > > > > > > > > > 0.9, recoveries are being throttled on a
> > > specific node in
> > > > > order
> > > > > > > to
> > > > > > > > > > reduce
> > > > > > > > > > > > > the load a that node. The throttling was
> > done on
> > > the node
> > > > > > > level,
> > > > > > > > > > after a
> > > > > > > > > > > > > shard has been allocated to it. Maintain
> > 
> > ```
> 
> the
> 
> > ```
> > > count of
> > > > > current
> > > > > > > > > > recoveries
> > > > > > > > > > > > is
> > > > > > > > > > > > > quite tricky because of the complexity of
> > 
> > ```
> 
> the
> 
> > ```
> > > recovery
> > > > > process.
> > > > > > > This
> > > > > > > > > > has
> > > > > > > > > > > > now
> > > > > > > > > > > > > been refactored into a better place, which
> > 
> > ```
> 
> is
> 
> > ```
> > > the actual
> > > > > > > > > > > > > allocation algorithm that runs and shuffles
> > > shards around.
> > > >
> > > > > > > > > > > > > The more interesting change is the
> > 
> > ```
> 
> gateway.
> 
> > ```
> > > There were
> > > > > > > several
> > > > > > > > > > > > problems
> > > > > > > > > > > > > with how the gateway works today that were
> > > exposed by a
> > > > > user of
> > > > > > > > > > > > > elasticsearch that stores 4TB data (several
> > > indices, each
> > > > > with
> > > > > > > 10
> > > > > > > > > > shards
> > > > > > > > > > > > and
> > > > > > > > > > > > > 2 replicas, which sums it up to 12TB).
> > This has
> > > uncovered
> > > > > some
> > > > > > > > > > problems
> > > > > > > > > > > > with
> > > > > > > > > > > > > the current design, specifically how md5
> > 
> > ```
> 
> are
> 
> > ```
> > > computed (and
> > > > > the
> > > > > > > time
> > > > > > > > > > it
> > > > > > > > > > > > takes
> > > > > > > > > > > > > to compute them on the local storage on
> > ec2), as
> > > well as
> > > > > other
> > > > > > > > > > > > possibilities
> > > > > > > > > > > > > for gateway corruptions using this load.
> > Of course,
> > > > > > > elasticsearch aim
> > > > > > > > > > is
> > > > > > > > > > > > to
> > > > > > > > > > > > > be able to store much more data than that,
> > 
> > ```
> 
> we
> 
> > ```
> > > are getting
> > > > > > > there... .
> > > >
> > > > > > > > > > > > > In general, the new implementation
> > works (in
> > > spirit) in
> > > > > the
> > > > > > > same
> > > > > > > > > > > > manner
> > > > > > > > > > > > > git works. Each snapshot is a commit
> > point, that
> > > stores
> > > > > files
> > > > > > > in the
> > > > > > > > > > > > gateway
> > > > > > > > > > > > > into an auto generated name, and finally, a
> > > commit point is
> > > > > > > written
> > > > > > > > > > with
> > > > > > > > > > > > the
> > > > > > > > > > > > > "directory" which maps between this pseudo
> > name
> > > to physical
> > > > > > > name, and
> > > > > > > > > > the
> > > > > > > > > > > > > size. The new design allows for more
> > resiliency
> > > when it
> > > > > comes
> > > > > > > to
> > > > > > > > > > > > corruption.
> > > > > > > > > > > > > It also allows for exciting future
> > features like
> > > saving a
> > > > > > > commit
> > > > > > > > > > point
> > > > > > > > > > > > and
> > > > > > > > > > > > > restoring from it, or automatically create
> > 
> > ```
> 
> a
> 
> > ```
> > > commit point
> > > > > each
> > > > > > > day
> > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > > last 5 days and be able to rollback to a
> > > specific commit
> > > > > point.
> > > >
> > > > > > > > > > > > > The aim is to create a gateway storage
> > 
> > ```
> 
> that
> 
> > ```
> > > is going to
> > > > > be
> > > > > > > the
> > > > > > > > > > final
> > > > > > > > > > > > > version, and resilient for future changes.
> > 
> > ```
> 
> It
> 
> > ```
> > > takes some
> > > > > time
> > > > > > > to get
> > > > > > > > > > > > there,
> > > > > > > > > > > > > but once we are there, I can safely stand
> > behind
> > > using
> > > > > > > elasticsearch
> > > > > > > > > > as
> > > > > > > > > > > > the
> > > > > > > > > > > > > main storage as well as releasing v1.0 (I
> > think
> > > > > elasticsearch
> > > > > > > has
> > > > > > > > > > enough
> > > > > > > > > > > > > features for 1.0, just the stability of the
> > > gateway is
> > > > > needed).
> > > >
> > > > > > > > > > > > > I would love for people to take this for
> > 
> > ```
> 
> a
> 
> > ```
> > > ride and
> > > > > check it
> > > > > > > out.
> > > > > > > > > > The
> > > > > > > > > > > > > next version, as a result of that is going
> > to be
> > > 0.10, and
> > > > > I
> > > > > > > will
> > > > > > > > > > release
> > > > > > > > > > > > it
> > > > > > > > > > > > > in the following days.
> > > >
> > > > > > > > > > > > > -shay.banon
> > >
> > >
> > 
> > ```

---

<div class="post-metadata">

**Author:** ![talsalmona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/talsalmona/32/3316_2.png) [@talsalmona](https://discuss.elastic.co/u/talsalmona)\
**Post date:** [August 29, 2010, 12:33pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245/18 "2010-08-29T12:33:08Z")

</div>

Hi Shay,  
Can you please explain or point me to the docs or example of how to  
work with the commit point API and how the commit / rollback workflow  
works?

Thanks,  
Tal

On Aug 26, 4:27 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:

> Yea, sure.
> 
> On Thu, Aug 26, 2010 at 3:30 PM, Kenneth Loafman  
> [kenneth.loaf...@gmail.com](mailto:kenneth.loaf...@gmail.com)wrote:
> 
> > Is it possible to copy a gateway from S3 to local, run the conversion  
> > locally, then copy it back to S3? The S3 rename issue could be solved  
> > this way, with less total downtime.
> 
> > ...Ken
> 
> > Shay Banon wrote:
> > 
> > > You can check what I do in the script, its pretty simple. The problem is  
> > > that S3 does not have a rename method..., though maybe a copy command  
> > > can be used, but not sure how long it takes for large files. You can  
> > > replace the file based operations with S3 ones.
> 
> > > -shay.banon
> 
> > > On Thu, Aug 26, 2010 at 2:22 AM, Kenneth Loafman  
> > > \<[kenneth.loaf...@gmail.com](mailto:kenneth.loaf...@gmail.com) [mailto:kenneth.loaf...@gmail.com](mailto:kenneth.loaf...@gmail.com)\> wrote:
> 
> > > ```
> > > Any hints for how to upgrade an S3 gateway?
> > > 
> > > ```
> 
> > > ```
> > > ...Ken
> > > 
> > > ```
> 
> > > ```
> > > Shay Banon wrote:
> > > > Yes, here it is:http://gist.github.com/546494.
> > > 
> > > ```
> 
> > > ```
> > > > -shay.banon
> > > 
> > > ```
> 
> > > ```
> > > > On Wed, Aug 25, 2010 at 8:46 PM, Paul <ppea...@gmail.com
> > > <mailto:ppea...@gmail.com>
> > > > <mailto:ppea...@gmail.com <mailto:ppea...@gmail.com>>> wrote:
> > > 
> > > ```
> 
> > > ```
> > > > Hi Shay,
> > > > Just curious, is the conversion script for file based gateway
> > > 
> > > ```
> > 
> > is
> > 
> > > ```
> > > > available? I haven't searched through GIT, so maybe it is
> > > 
> > > ```
> > 
> > there?
> 
> > > ```
> > > > Spent the weekend building up 20mil docs and would prefer not
> > > 
> > > ```
> > 
> > to
> > 
> > > ```
> > > > repeat this.
> > > 
> > > ```
> 
> > > ```
> > > > Thanks,
> > > > Paul
> > > 
> > > ```
> 
> > > ```
> > > > On Aug 24, 4:34 pm, Shay Banon <shay.ba...@elasticsearch.com
> > > <mailto:shay.ba...@elasticsearch.com>
> > > > <mailto:shay.ba...@elasticsearch.com
> > > <mailto:shay.ba...@elasticsearch.com>>> wrote:
> > > > > Indeed. All the __xxx fils are binary files (either index
> > > files or
> > > > > transaction log parts). The commit-N is a json file that
> > > provides meta
> > > > > information on the commit point.
> > > 
> > > ```
> 
> > > ```
> > > > > On Wed, Aug 25, 2010 at 1:32 AM, Grant Rodgers
> > > <gra...@gmail.com <mailto:gra...@gmail.com>
> > > > <mailto:gra...@gmail.com <mailto:gra...@gmail.com>>> wrote:
> > > > > > I was very curious to see what the file format was!
> > > 
> > > ```
> 
> > > ```
> > > > > > On Aug 24, 3:05 pm, Shay Banon
> > > <shay.ba...@elasticsearch.com <mailto:shay.ba...@elasticsearch.com>
> > > > <mailto:shay.ba...@elasticsearch.com
> > > <mailto:shay.ba...@elasticsearch.com>>> wrote:
> > > > > > > Great, thanks for validating that. Did not expect someone
> > > 
> > > ```
> > 
> > to
> > 
> > > ```
> > > > open vi on
> > > > > > the
> > > > > > > gateway ;)
> > > 
> > > ```
> 
> > > ```
> > > > > > > On Wed, Aug 25, 2010 at 12:18 AM, Grant Rodgers
> > > > <gra...@gmail.com <mailto:gra...@gmail.com>
> > > <mailto:gra...@gmail.com <mailto:gra...@gmail.com>>>
> > > > > > wrote:
> > > > > > > > I think it's fixed in head; I didn't see this error
> > > 
> > > ```
> > 
> > again
> > 
> > > ```
> > > > when trying
> > > > > > > > the test below.
> > > 
> > > ```
> 
> > > ```
> > > > > > > > On Aug 24, 1:12 pm, Grant Rodgers <gra...@gmail.com
> > > <mailto:gra...@gmail.com>
> > > > <mailto:gra...@gmail.com <mailto:gra...@gmail.com>>> wrote:
> > > > > > > > > Oh you know it was probably a vi swap file. I was
> > > taking a
> > > > look at
> > > > > > one
> > > > > > > > > of the commit logs, and it might have tried to
> > > 
> > > ```
> > 
> > snapshot
> > 
> > > ```
> > > > while it was
> > > > > > > > > open.
> > > 
> > > ```
> 
> > > ```
> > > > > > > > > I think you have committed another change since
> > > 
> > > ```
> > 
> > 80c7135
> > 
> > > ```
> > > > that ignores
> > > > > > > > > files elasticsearch didn't create. I'll build the
> > > 
> > > ```
> > 
> > latest
> > 
> > > ```
> > > > head and try
> > > > > > > > > viewing a commit snapshot again.
> > > 
> > > ```
> 
> > > ```
> > > > > > > > > On Aug 24, 1:00 pm, Shay Banon
> > > > <shay.ba...@elasticsearch.com
> > > <mailto:shay.ba...@elasticsearch.com>
> > > <mailto:shay.ba...@elasticsearch.com
> > > <mailto:shay.ba...@elasticsearch.com>>>
> > > > wrote:
> > > 
> > > ```
> 
> > > ```
> > > > > > > > > > Strange, not sure how this file ended up in the
> > > gateway,
> > > > I can't
> > > > > > see
> > > > > > > > where
> > > > > > > > > > elasticsearch would write it. It only writes __xxx
> > > files
> > > > (no .
> > > > > > > > something)
> > > > > > > > > > and commit- files. I will fix it to ignore files
> > > 
> > > ```
> > 
> > that
> > 
> > > ```
> > > > don't conform
> > > > > > to
> > > > > > > > the
> > > > > > > > > > format, but we should try and understand where its
> > > > coming from...
> > > 
> > > ```
> 
> > > ```
> > > > > > > > > > -shay.banon
> > > 
> > > ```
> 
> > > ```
> > > > > > > > > > On Tue, Aug 24, 2010 at 10:25 PM, Grant Rodgers
> > > > <gra...@gmail.com <mailto:gra...@gmail.com>
> > > <mailto:gra...@gmail.com <mailto:gra...@gmail.com>>>
> > > > > > > > wrote:
> > > > > > > > > > > Oh btw that error was with 80c7135
> > > 
> > > ```
> 
> > > ```
> > > > > > > > > > > On Aug 23, 12:22 pm, Shay Banon
> > > > <shay.ba...@elasticsearch.com
> > > <mailto:shay.ba...@elasticsearch.com>
> > > <mailto:shay.ba...@elasticsearch.com
> > > <mailto:shay.ba...@elasticsearch.com>>>
> > > > > > > > wrote:
> > > > > > > > > > > > Yes, though there is a whole "workflow" level
> > > 
> > > ```
> > 
> > APIs
> > 
> > > ```
> > > > and support
> > > > > > for.
> > > > > > > > But,
> > > > > > > > > > > the
> > > > > > > > > > > > basics are there in the gateway.
> > > 
> > > ```
> 
> > > ```
> > > > > > > > > > > > -shay.banon
> > > 
> > > ```
> 
> > > ```
> > > > > > > > > > > > On Mon, Aug 23, 2010 at 10:21 PM, Tal
> > > > <talsalm...@gmail.com <mailto:talsalm...@gmail.com>
> > > <mailto:talsalm...@gmail.com <mailto:talsalm...@gmail.com>>>
> > > > > > > > wrote:
> > > > > > > > > > > > > Very nice feature indeed.
> > > > > > > > > > > > > Will this also allow for index level commit
> > > points?
> > > 
> > > ```
> 
> > > ```
> > > > > > > > > > > > > Tal
> > > 
> > > ```
> 
> > > ```
> > > > > > > > > > > > > On Aug 23, 10:11 pm, Shay Banon <
> > > > > > shay.ba...@elasticsearch.com
> > > <mailto:shay.ba...@elasticsearch.com>
> > > <mailto:shay.ba...@elasticsearch.com
> > > <mailto:shay.ba...@elasticsearch.com>>>
> > > > > > > > wrote:
> > > > > > > > > > > > > > Hi,
> > > 
> > > ```
> 
> > > ```
> > > > > > > > > > > > > > I am going to push (pretty soon) a
> > > 
> > > ```
> > 
> > major
> > 
> > > ```
> > > > rewrite of the
> > > > > > > > gateway
> > > > > > > > > > > > > module
> > > > > > > > > > > > > > and improved throttling support. The
> > > 
> > > ```
> > 
> > gateway
> > 
> > > ```
> > > > change is a
> > > > > > > > breaking
> > > > > > > > > > > change,
> > > > > > > > > > > > > > meaning that new version will not be able
> > > 
> > > ```
> > 
> > to
> > 
> > > ```
> > > > recover from a
> > > > > > 0.9
> > > > > > > > > > > gateway.
> > > > > > > > > > > > > I
> > > > > > > > > > > > > > will provide an upgrade script for file
> > > 
> > > ```
> > 
> > based
> > 
> > > ```
> > > > gateway, s3
> > > > > > based
> > > > > > > > > > > gateway
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > require reindexing, though, potentially,
> > > 
> > > ```
> > 
> > that
> > 
> > > ```
> > > > script can be
> > > > > > > > adjusted
> > > > > > > > > > > to
> > > > > > > > > > > > > > support it.
> > > 
> > > ```
> 
> > > ```
> > > > > > > > > > > > > > Let me explain some of the changes. The
> > > first is
> > > > > > throttling
> > > > > > > > > > > support.
> > > > > > > > > > > > > In
> > > > > > > > > > > > > > 0.9, recoveries are being throttled on a
> > > > specific node in
> > > > > > order
> > > > > > > > to
> > > > > > > > > > > reduce
> > > > > > > > > > > > > > the load a that node. The throttling was
> > > done on
> > > > the node
> > > > > > > > level,
> > > > > > > > > > > after a
> > > > > > > > > > > > > > shard has been allocated to it. Maintain
> > > 
> > > ```
> > 
> > the
> > 
> > > ```
> > > > count of
> > > > > > current
> > > > > > > > > > > recoveries
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > quite tricky because of the complexity of
> > > 
> > > ```
> > 
> > the
> > 
> > > ```
> > > > recovery
> > > > > > process.
> > > > > > > > This
> > > > > > > > > > > has
> > > > > > > > > > > > > now
> > > > > > > > > > > > > > been refactored into a better place, which
> > > 
> > > ```
> > 
> > is
> > 
> > > ```
> > > > the actual
> > > > > > > > > > > > > > allocation algorithm that runs and shuffles
> > > > shards around.
> > > 
> > > ```
> 
> > > ```
> > > > > > > > > > > > > > The more interesting change is the
> > > 
> > > ```
> > 
> > gateway.
> > 
> > > ```
> > > > There were
> > > > > > > > several
> > > > > > > > > > > > > problems
> > > > > > > > > > > > > > with how the gateway works today that were
> > > > exposed by a
> > > > > > user of
> > > > > > > > > > > > > > elasticsearch that stores 4TB data (several
> > > > indices, each
> > > > > > with
> > > > > > > > 10
> > > > > > > > > > > shards
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > 2 replicas, which sums it up to 12TB).
> > > This has
> > > > uncovered
> > > > > > some
> > > > > > > > > > > problems
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > the current design, specifically how md5
> > > 
> > > ```
> > 
> > are
> > 
> > > ```
> > > > computed (and
> > > > > > the
> > > > > > > > time
> > > > > > > > > > > it
> > > > > > > > > > > > > takes
> > > > > > > > > > > > > > to compute them on the local storage on
> > > ec2), as
> > > > well as
> > > > > > other
> > > > > > > > > > > > > possibilities
> > > > > > > > > > > > > > for gateway corruptions using this load.
> > > Of course,
> > > > > > > > elasticsearch aim
> > > > > > > > > > > is
> > > > > > > > > > > > > to
> > > 
> > > ```
> 
> ...
> 
> read more »

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 29, 2010, 12:54pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245/19 "2010-08-29T12:54:09Z")

</div>

There isn't an API for that. Thats what I meant that there is a whole  
"orchestration" API that needs to be built on top of the new capabilities of  
the gateway storage model. The good news is that the new storage model  
allows for that, but there is still work left to actually expose it to the  
user.

-shay.banon

On Sun, Aug 29, 2010 at 3:33 PM, Tal [talsalmona@gmail.com](mailto:talsalmona@gmail.com) wrote:

> Hi Shay,  
> Can you please explain or point me to the docs or example of how to  
> work with the commit point API and how the commit / rollback workflow  
> works?
> 
> Thanks,  
> Tal
> 
> On Aug 26, 4:27 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> 
> > Yea, sure.
> > 
> > On Thu, Aug 26, 2010 at 3:30 PM, Kenneth Loafman  
> > [kenneth.loaf...@gmail.com](mailto:kenneth.loaf...@gmail.com)wrote:
> > 
> > > Is it possible to copy a gateway from S3 to local, run the conversion  
> > > locally, then copy it back to S3? The S3 rename issue could be solved  
> > > this way, with less total downtime.
> > 
> > > ...Ken
> > 
> > > Shay Banon wrote:
> > > 
> > > > You can check what I do in the script, its pretty simple. The problem  
> > > > is  
> > > > that S3 does not have a rename method..., though maybe a copy command  
> > > > can be used, but not sure how long it takes for large files. You can  
> > > > replace the file based operations with S3 ones.
> > 
> > > > -shay.banon
> > 
> > > > On Thu, Aug 26, 2010 at 2:22 AM, Kenneth Loafman  
> > > > \<[kenneth.loaf...@gmail.com](mailto:kenneth.loaf...@gmail.com) [mailto:kenneth.loaf...@gmail.com](mailto:kenneth.loaf...@gmail.com)\>  
> > > > wrote:
> > 
> > > > ```
> > > > Any hints for how to upgrade an S3 gateway?
> > > > 
> > > > ```
> > 
> > > > ```
> > > > ...Ken
> > > > 
> > > > ```
> > 
> > > > ```
> > > > Shay Banon wrote:
> > > > > Yes, here it is:http://gist.github.com/546494.
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > -shay.banon
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > On Wed, Aug 25, 2010 at 8:46 PM, Paul <ppea...@gmail.com
> > > > <mailto:ppea...@gmail.com>
> > > > > <mailto:ppea...@gmail.com <mailto:ppea...@gmail.com>>> wrote:
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > Hi Shay,
> > > > > Just curious, is the conversion script for file based
> > > > 
> > > > ```
> 
> gateway
> 
> > > is
> > > 
> > > > ```
> > > > > available? I haven't searched through GIT, so maybe it is
> > > > 
> > > > ```
> > > 
> > > there?
> > 
> > > > ```
> > > > > Spent the weekend building up 20mil docs and would prefer
> > > > 
> > > > ```
> 
> not
> 
> > > to
> > > 
> > > > ```
> > > > > repeat this.
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > Thanks,
> > > > > Paul
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > On Aug 24, 4:34 pm, Shay Banon <
> > > > 
> > > > ```
> 
> [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)
> 
> > > > ```
> > > > <mailto:shay.ba...@elasticsearch.com>
> > > > > <mailto:shay.ba...@elasticsearch.com
> > > > <mailto:shay.ba...@elasticsearch.com>>> wrote:
> > > > > > Indeed. All the __xxx fils are binary files (either index
> > > > files or
> > > > > > transaction log parts). The commit-N is a json file that
> > > > provides meta
> > > > > > information on the commit point.
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > > On Wed, Aug 25, 2010 at 1:32 AM, Grant Rodgers
> > > > <gra...@gmail.com <mailto:gra...@gmail.com>
> > > > > <mailto:gra...@gmail.com <mailto:gra...@gmail.com>>>
> > > > 
> > > > ```
> 
> wrote:
> 
> > > > ```
> > > > > > > I was very curious to see what the file format was!
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > > > On Aug 24, 3:05 pm, Shay Banon
> > > > <shay.ba...@elasticsearch.com <mailto:
> > > > 
> > > > ```
> 
> [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)\>
> 
> > > > ```
> > > > > <mailto:shay.ba...@elasticsearch.com
> > > > <mailto:shay.ba...@elasticsearch.com>>> wrote:
> > > > > > > > Great, thanks for validating that. Did not expect
> > > > 
> > > > ```
> 
> someone
> 
> > > to
> > > 
> > > > ```
> > > > > open vi on
> > > > > > > the
> > > > > > > > gateway ;)
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > > > > On Wed, Aug 25, 2010 at 12:18 AM, Grant Rodgers
> > > > > <gra...@gmail.com <mailto:gra...@gmail.com>
> > > > <mailto:gra...@gmail.com <mailto:gra...@gmail.com>>>
> > > > > > > wrote:
> > > > > > > > > I think it's fixed in head; I didn't see this error
> > > > 
> > > > ```
> > > 
> > > again
> > > 
> > > > ```
> > > > > when trying
> > > > > > > > > the test below.
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > > > > > On Aug 24, 1:12 pm, Grant Rodgers <
> > > > 
> > > > ```
> 
> [gra...@gmail.com](mailto:gra...@gmail.com)
> 
> > > > ```
> > > > <mailto:gra...@gmail.com>
> > > > > <mailto:gra...@gmail.com <mailto:gra...@gmail.com>>>
> > > > 
> > > > ```
> 
> wrote:
> 
> > > > ```
> > > > > > > > > > Oh you know it was probably a vi swap file. I was
> > > > taking a
> > > > > look at
> > > > > > > one
> > > > > > > > > > of the commit logs, and it might have tried to
> > > > 
> > > > ```
> > > 
> > > snapshot
> > > 
> > > > ```
> > > > > while it was
> > > > > > > > > > open.
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > > > > > > I think you have committed another change since
> > > > 
> > > > ```
> > > 
> > > 80c7135
> > > 
> > > > ```
> > > > > that ignores
> > > > > > > > > > files elasticsearch didn't create. I'll build the
> > > > 
> > > > ```
> > > 
> > > latest
> > > 
> > > > ```
> > > > > head and try
> > > > > > > > > > viewing a commit snapshot again.
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > > > > > > On Aug 24, 1:00 pm, Shay Banon
> > > > > <shay.ba...@elasticsearch.com
> > > > <mailto:shay.ba...@elasticsearch.com>
> > > > <mailto:shay.ba...@elasticsearch.com
> > > > <mailto:shay.ba...@elasticsearch.com>>>
> > > > > wrote:
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > > > > > > > Strange, not sure how this file ended up in the
> > > > gateway,
> > > > > I can't
> > > > > > > see
> > > > > > > > > where
> > > > > > > > > > > elasticsearch would write it. It only writes
> > > > 
> > > > ```
> 
> \_\_xxx
> 
> > > > ```
> > > > files
> > > > > (no .
> > > > > > > > > something)
> > > > > > > > > > > and commit- files. I will fix it to ignore
> > > > 
> > > > ```
> 
> files
> 
> > > that
> > > 
> > > > ```
> > > > > don't conform
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > format, but we should try and understand where
> > > > 
> > > > ```
> 
> its
> 
> > > > ```
> > > > > coming from...
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > > > > > > > -shay.banon
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > > > > > > > On Tue, Aug 24, 2010 at 10:25 PM, Grant Rodgers
> > > > > <gra...@gmail.com <mailto:gra...@gmail.com>
> > > > <mailto:gra...@gmail.com <mailto:gra...@gmail.com>>>
> > > > > > > > > wrote:
> > > > > > > > > > > > Oh btw that error was with 80c7135
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > > > > > > > > On Aug 23, 12:22 pm, Shay Banon
> > > > > <shay.ba...@elasticsearch.com
> > > > <mailto:shay.ba...@elasticsearch.com>
> > > > <mailto:shay.ba...@elasticsearch.com
> > > > <mailto:shay.ba...@elasticsearch.com>>>
> > > > > > > > > wrote:
> > > > > > > > > > > > > Yes, though there is a whole "workflow"
> > > > 
> > > > ```
> 
> level
> 
> > > APIs
> > > 
> > > > ```
> > > > > and support
> > > > > > > for.
> > > > > > > > > But,
> > > > > > > > > > > > the
> > > > > > > > > > > > > basics are there in the gateway.
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > > > > > > > > > -shay.banon
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > > > > > > > > > On Mon, Aug 23, 2010 at 10:21 PM, Tal
> > > > > <talsalm...@gmail.com <mailto:talsalm...@gmail.com>
> > > > <mailto:talsalm...@gmail.com <mailto:talsalm...@gmail.com>>>
> > > > > > > > > wrote:
> > > > > > > > > > > > > > Very nice feature indeed.
> > > > > > > > > > > > > > Will this also allow for index level
> > > > 
> > > > ```
> 
> commit
> 
> > > > ```
> > > > points?
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > > > > > > > > > > Tal
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > > > > > > > > > > On Aug 23, 10:11 pm, Shay Banon <
> > > > > > > shay.ba...@elasticsearch.com
> > > > <mailto:shay.ba...@elasticsearch.com>
> > > > <mailto:shay.ba...@elasticsearch.com
> > > > <mailto:shay.ba...@elasticsearch.com>>>
> > > > > > > > > wrote:
> > > > > > > > > > > > > > > Hi,
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > > > > > > > > > > > I am going to push (pretty soon) a
> > > > 
> > > > ```
> > > 
> > > major
> > > 
> > > > ```
> > > > > rewrite of the
> > > > > > > > > gateway
> > > > > > > > > > > > > > module
> > > > > > > > > > > > > > > and improved throttling support. The
> > > > 
> > > > ```
> > > 
> > > gateway
> > > 
> > > > ```
> > > > > change is a
> > > > > > > > > breaking
> > > > > > > > > > > > change,
> > > > > > > > > > > > > > > meaning that new version will not be
> > > > 
> > > > ```
> 
> able
> 
> > > to
> > > 
> > > > ```
> > > > > recover from a
> > > > > > > 0.9
> > > > > > > > > > > > gateway.
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > will provide an upgrade script for file
> > > > 
> > > > ```
> > > 
> > > based
> > > 
> > > > ```
> > > > > gateway, s3
> > > > > > > based
> > > > > > > > > > > > gateway
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > require reindexing, though,
> > > > 
> > > > ```
> 
> potentially,
> 
> > > that
> > > 
> > > > ```
> > > > > script can be
> > > > > > > > > adjusted
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > support it.
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > > > > > > > > > > > Let me explain some of the changes.
> > > > 
> > > > ```
> 
> The
> 
> > > > ```
> > > > first is
> > > > > > > throttling
> > > > > > > > > > > > support.
> > > > > > > > > > > > > > In
> > > > > > > > > > > > > > > 0.9, recoveries are being throttled on
> > > > 
> > > > ```
> 
> a
> 
> > > > ```
> > > > > specific node in
> > > > > > > order
> > > > > > > > > to
> > > > > > > > > > > > reduce
> > > > > > > > > > > > > > > the load a that node. The throttling
> > > > 
> > > > ```
> 
> was
> 
> > > > ```
> > > > done on
> > > > > the node
> > > > > > > > > level,
> > > > > > > > > > > > after a
> > > > > > > > > > > > > > > shard has been allocated to it.
> > > > 
> > > > ```
> 
> Maintain
> 
> > > the
> > > 
> > > > ```
> > > > > count of
> > > > > > > current
> > > > > > > > > > > > recoveries
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > quite tricky because of the complexity
> > > > 
> > > > ```
> 
> of
> 
> > > the
> > > 
> > > > ```
> > > > > recovery
> > > > > > > process.
> > > > > > > > > This
> > > > > > > > > > > > has
> > > > > > > > > > > > > > now
> > > > > > > > > > > > > > > been refactored into a better place,
> > > > 
> > > > ```
> 
> which
> 
> > > is
> > > 
> > > > ```
> > > > > the actual
> > > > > > > > > > > > > > > allocation algorithm that runs and
> > > > 
> > > > ```
> 
> shuffles
> 
> > > > ```
> > > > > shards around.
> > > > 
> > > > ```
> > 
> > > > ```
> > > > > > > > > > > > > > > The more interesting change is the
> > > > 
> > > > ```
> > > 
> > > gateway.
> > > 
> > > > ```
> > > > > There were
> > > > > > > > > several
> > > > > > > > > > > > > > problems
> > > > > > > > > > > > > > > with how the gateway works today that
> > > > 
> > > > ```
> 
> were
> 
> > > > ```
> > > > > exposed by a
> > > > > > > user of
> > > > > > > > > > > > > > > elasticsearch that stores 4TB data
> > > > 
> > > > ```
> 
> (several
> 
> > > > ```
> > > > > indices, each
> > > > > > > with
> > > > > > > > > 10
> > > > > > > > > > > > shards
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > 2 replicas, which sums it up to 12TB).
> > > > This has
> > > > > uncovered
> > > > > > > some
> > > > > > > > > > > > problems
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > the current design, specifically how
> > > > 
> > > > ```
> 
> md5
> 
> > > are
> > > 
> > > > ```
> > > > > computed (and
> > > > > > > the
> > > > > > > > > time
> > > > > > > > > > > > it
> > > > > > > > > > > > > > takes
> > > > > > > > > > > > > > > to compute them on the local storage on
> > > > ec2), as
> > > > > well as
> > > > > > > other
> > > > > > > > > > > > > > possibilities
> > > > > > > > > > > > > > > for gateway corruptions using this
> > > > 
> > > > ```
> 
> load.
> 
> > > > ```
> > > > Of course,
> > > > > > > > > elasticsearch aim
> > > > > > > > > > > > is
> > > > > > > > > > > > > > to
> > > > 
> > > > ```
> > 
> > ...
> > 
> > read more »

---

<div class="post-metadata">

**Author:** ![talsalmona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/talsalmona/32/3316_2.png) [@talsalmona](https://discuss.elastic.co/u/talsalmona)\
**Post date:** [August 29, 2010, 1:07pm UTC](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245/20 "2010-08-29T13:07:48Z")

</div>

I see. Looking forward for it 🙂

Thanks,  
Tal

On Aug 29, 3:54 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:

> There isn't an API for that. Thats what I meant that there is a whole  
> "orchestration" API that needs to be built on top of the new capabilities of  
> the gateway storage model. The good news is that the new storage model  
> allows for that, but there is still work left to actually expose it to the  
> user.
> 
> -shay.banon
> 
> On Sun, Aug 29, 2010 at 3:33 PM, Tal [talsalm...@gmail.com](mailto:talsalm...@gmail.com) wrote:
> 
> > Hi Shay,  
> > Can you please explain or point me to the docs or example of how to  
> > work with the commit point API and how the commit / rollback workflow  
> > works?
> 
> > Thanks,  
> > Tal
> 
> > On Aug 26, 4:27 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> > 
> > > Yea, sure.
> 
> > > On Thu, Aug 26, 2010 at 3:30 PM, Kenneth Loafman  
> > > [kenneth.loaf...@gmail.com](mailto:kenneth.loaf...@gmail.com)wrote:
> 
> > > > Is it possible to copy a gateway from S3 to local, run the conversion  
> > > > locally, then copy it back to S3? The S3 rename issue could be solved  
> > > > this way, with less total downtime.
> 
> > > > ...Ken
> 
> > > > Shay Banon wrote:
> > > > 
> > > > > You can check what I do in the script, its pretty simple. The problem  
> > > > > is  
> > > > > that S3 does not have a rename method..., though maybe a copy command  
> > > > > can be used, but not sure how long it takes for large files. You can  
> > > > > replace the file based operations with S3 ones.
> 
> > > > > -shay.banon
> 
> > > > > On Thu, Aug 26, 2010 at 2:22 AM, Kenneth Loafman  
> > > > > \<[kenneth.loaf...@gmail.com](mailto:kenneth.loaf...@gmail.com) [mailto:kenneth.loaf...@gmail.com](mailto:kenneth.loaf...@gmail.com)\>  
> > > > > wrote:
> 
> > > > > ```
> > > > > Any hints for how to upgrade an S3 gateway?
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > ...Ken
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > Shay Banon wrote:
> > > > > > Yes, here it is:http://gist.github.com/546494.
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > -shay.banon
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > On Wed, Aug 25, 2010 at 8:46 PM, Paul <ppea...@gmail.com
> > > > > <mailto:ppea...@gmail.com>
> > > > > > <mailto:ppea...@gmail.com <mailto:ppea...@gmail.com>>> wrote:
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > Hi Shay,
> > > > > > Just curious, is the conversion script for file based
> > > > > 
> > > > > ```
> > 
> > gateway
> > 
> > > > is
> > > > 
> > > > > ```
> > > > > > available? I haven't searched through GIT, so maybe it is
> > > > > 
> > > > > ```
> > > > 
> > > > there?
> 
> > > > > ```
> > > > > > Spent the weekend building up 20mil docs and would prefer
> > > > > 
> > > > > ```
> > 
> > not
> > 
> > > > to
> > > > 
> > > > > ```
> > > > > > repeat this.
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > Thanks,
> > > > > > Paul
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > On Aug 24, 4:34 pm, Shay Banon <
> > > > > 
> > > > > ```
> > 
> > [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)
> > 
> > > > > ```
> > > > > <mailto:shay.ba...@elasticsearch.com>
> > > > > > <mailto:shay.ba...@elasticsearch.com
> > > > > <mailto:shay.ba...@elasticsearch.com>>> wrote:
> > > > > > > Indeed. All the __xxx fils are binary files (either index
> > > > > files or
> > > > > > > transaction log parts). The commit-N is a json file that
> > > > > provides meta
> > > > > > > information on the commit point.
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > > On Wed, Aug 25, 2010 at 1:32 AM, Grant Rodgers
> > > > > <gra...@gmail.com <mailto:gra...@gmail.com>
> > > > > > <mailto:gra...@gmail.com <mailto:gra...@gmail.com>>>
> > > > > 
> > > > > ```
> > 
> > wrote:
> > 
> > > > > ```
> > > > > > > > I was very curious to see what the file format was!
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > > > On Aug 24, 3:05 pm, Shay Banon
> > > > > <shay.ba...@elasticsearch.com <mailto:
> > > > > 
> > > > > ```
> > 
> > [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)\>
> > 
> > > > > ```
> > > > > > <mailto:shay.ba...@elasticsearch.com
> > > > > <mailto:shay.ba...@elasticsearch.com>>> wrote:
> > > > > > > > > Great, thanks for validating that. Did not expect
> > > > > 
> > > > > ```
> > 
> > someone
> > 
> > > > to
> > > > 
> > > > > ```
> > > > > > open vi on
> > > > > > > > the
> > > > > > > > > gateway ;)
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > > > > On Wed, Aug 25, 2010 at 12:18 AM, Grant Rodgers
> > > > > > <gra...@gmail.com <mailto:gra...@gmail.com>
> > > > > <mailto:gra...@gmail.com <mailto:gra...@gmail.com>>>
> > > > > > > > wrote:
> > > > > > > > > > I think it's fixed in head; I didn't see this error
> > > > > 
> > > > > ```
> > > > 
> > > > again
> > > > 
> > > > > ```
> > > > > > when trying
> > > > > > > > > > the test below.
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > > > > > On Aug 24, 1:12 pm, Grant Rodgers <
> > > > > 
> > > > > ```
> > 
> > [gra...@gmail.com](mailto:gra...@gmail.com)
> > 
> > > > > ```
> > > > > <mailto:gra...@gmail.com>
> > > > > > <mailto:gra...@gmail.com <mailto:gra...@gmail.com>>>
> > > > > 
> > > > > ```
> > 
> > wrote:
> > 
> > > > > ```
> > > > > > > > > > > Oh you know it was probably a vi swap file. I was
> > > > > taking a
> > > > > > look at
> > > > > > > > one
> > > > > > > > > > > of the commit logs, and it might have tried to
> > > > > 
> > > > > ```
> > > > 
> > > > snapshot
> > > > 
> > > > > ```
> > > > > > while it was
> > > > > > > > > > > open.
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > > > > > > I think you have committed another change since
> > > > > 
> > > > > ```
> > > > 
> > > > 80c7135
> > > > 
> > > > > ```
> > > > > > that ignores
> > > > > > > > > > > files elasticsearch didn't create. I'll build the
> > > > > 
> > > > > ```
> > > > 
> > > > latest
> > > > 
> > > > > ```
> > > > > > head and try
> > > > > > > > > > > viewing a commit snapshot again.
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > > > > > > On Aug 24, 1:00 pm, Shay Banon
> > > > > > <shay.ba...@elasticsearch.com
> > > > > <mailto:shay.ba...@elasticsearch.com>
> > > > > <mailto:shay.ba...@elasticsearch.com
> > > > > <mailto:shay.ba...@elasticsearch.com>>>
> > > > > > wrote:
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > > > > > > > Strange, not sure how this file ended up in the
> > > > > gateway,
> > > > > > I can't
> > > > > > > > see
> > > > > > > > > > where
> > > > > > > > > > > > elasticsearch would write it. It only writes
> > > > > 
> > > > > ```
> > 
> > \_\_xxx
> > 
> > > > > ```
> > > > > files
> > > > > > (no .
> > > > > > > > > > something)
> > > > > > > > > > > > and commit- files. I will fix it to ignore
> > > > > 
> > > > > ```
> > 
> > files
> > 
> > > > that
> > > > 
> > > > > ```
> > > > > > don't conform
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > format, but we should try and understand where
> > > > > 
> > > > > ```
> > 
> > its
> > 
> > > > > ```
> > > > > > coming from...
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > > > > > > > -shay.banon
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > > > > > > > On Tue, Aug 24, 2010 at 10:25 PM, Grant Rodgers
> > > > > > <gra...@gmail.com <mailto:gra...@gmail.com>
> > > > > <mailto:gra...@gmail.com <mailto:gra...@gmail.com>>>
> > > > > > > > > > wrote:
> > > > > > > > > > > > > Oh btw that error was with 80c7135
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > > > > > > > > On Aug 23, 12:22 pm, Shay Banon
> > > > > > <shay.ba...@elasticsearch.com
> > > > > <mailto:shay.ba...@elasticsearch.com>
> > > > > <mailto:shay.ba...@elasticsearch.com
> > > > > <mailto:shay.ba...@elasticsearch.com>>>
> > > > > > > > > > wrote:
> > > > > > > > > > > > > > Yes, though there is a whole "workflow"
> > > > > 
> > > > > ```
> > 
> > level
> > 
> > > > APIs
> > > > 
> > > > > ```
> > > > > > and support
> > > > > > > > for.
> > > > > > > > > > But,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > basics are there in the gateway.
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > > > > > > > > > -shay.banon
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > > > > > > > > > On Mon, Aug 23, 2010 at 10:21 PM, Tal
> > > > > > <talsalm...@gmail.com <mailto:talsalm...@gmail.com>
> > > > > <mailto:talsalm...@gmail.com <mailto:talsalm...@gmail.com>>>
> > > > > > > > > > wrote:
> > > > > > > > > > > > > > > Very nice feature indeed.
> > > > > > > > > > > > > > > Will this also allow for index level
> > > > > 
> > > > > ```
> > 
> > commit
> > 
> > > > > ```
> > > > > points?
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > > > > > > > > > > Tal
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > > > > > > > > > > On Aug 23, 10:11 pm, Shay Banon <
> > > > > > > > shay.ba...@elasticsearch.com
> > > > > <mailto:shay.ba...@elasticsearch.com>
> > > > > <mailto:shay.ba...@elasticsearch.com
> > > > > <mailto:shay.ba...@elasticsearch.com>>>
> > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > Hi,
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > > > > > > > > > > > I am going to push (pretty soon) a
> > > > > 
> > > > > ```
> > > > 
> > > > major
> > > > 
> > > > > ```
> > > > > > rewrite of the
> > > > > > > > > > gateway
> > > > > > > > > > > > > > > module
> > > > > > > > > > > > > > > > and improved throttling support. The
> > > > > 
> > > > > ```
> > > > 
> > > > gateway
> > > > 
> > > > > ```
> > > > > > change is a
> > > > > > > > > > breaking
> > > > > > > > > > > > > change,
> > > > > > > > > > > > > > > > meaning that new version will not be
> > > > > 
> > > > > ```
> > 
> > able
> > 
> > > > to
> > > > 
> > > > > ```
> > > > > > recover from a
> > > > > > > > 0.9
> > > > > > > > > > > > > gateway.
> > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > will provide an upgrade script for file
> > > > > 
> > > > > ```
> > > > 
> > > > based
> > > > 
> > > > > ```
> > > > > > gateway, s3
> > > > > > > > based
> > > > > > > > > > > > > gateway
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > require reindexing, though,
> > > > > 
> > > > > ```
> > 
> > potentially,
> > 
> > > > that
> > > > 
> > > > > ```
> > > > > > script can be
> > > > > > > > > > adjusted
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > support it.
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > > > > > > > > > > > Let me explain some of the changes.
> > > > > 
> > > > > ```
> > 
> > The
> > 
> > > > > ```
> > > > > first is
> > > > > > > > throttling
> > > > > > > > > > > > > support.
> > > > > > > > > > > > > > > In
> > > > > > > > > > > > > > > > 0.9, recoveries are being throttled on
> > > > > 
> > > > > ```
> > 
> > a
> > 
> > > > > ```
> > > > > > specific node in
> > > > > > > > order
> > > > > > > > > > to
> > > > > > > > > > > > > reduce
> > > > > > > > > > > > > > > > the load a that node. The throttling
> > > > > 
> > > > > ```
> > 
> > was
> > 
> > > > > ```
> > > > > done on
> > > > > > the node
> > > > > > > > > > level,
> > > > > > > > > > > > > after a
> > > > > > > > > > > > > > > > shard has been allocated to it.
> > > > > 
> > > > > ```
> > 
> > Maintain
> > 
> > > > the
> > > > 
> > > > > ```
> > > > > > count of
> > > > > > > > current
> > > > > > > > > > > > > recoveries
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > quite tricky because of the complexity
> > > > > 
> > > > > ```
> > 
> > of
> > 
> > > > the
> > > > 
> > > > > ```
> > > > > > recovery
> > > > > > > > process.
> > > > > > > > > > This
> > > > > > > > > > > > > has
> > > > > > > > > > > > > > > now
> > > > > > > > > > > > > > > > been refactored into a better place,
> > > > > 
> > > > > ```
> > 
> > which
> > 
> > > > is
> > > > 
> > > > > ```
> > > > > > the actual
> > > > > > > > > > > > > > > > allocation algorithm that runs and
> > > > > 
> > > > > ```
> > 
> > shuffles
> > 
> > > > > ```
> > > > > > shards around.
> > > > > 
> > > > > ```
> 
> > > > > ```
> > > > > > > > > > > > > > > > The more interesting change is the
> > > > > 
> > > > > ```
> > > > 
> > > > gateway.
> > > > 
> > > > > ```
> > > > > > There were
> > > > > 
> > > > > ```
> 
> ...
> 
> read more »

[Next page](https://discuss.elastic.co/t/breaking-change-majore-gateway-refactoring-and-improved-throttling/3245.md?page=2)