# Impact of using S3 Gateway

**URL:** <https://discuss.elastic.co/t/impact-of-using-s3-gateway/5213>\
**Category:** Elasticsearch\
**Created:** [August 21, 2011, 8:26pm UTC](https://discuss.elastic.co/t/impact-of-using-s3-gateway/5213 "2011-08-21T20:26:34Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![James\_Cook](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@James\_Cook](https://discuss.elastic.co/u/James_Cook)\
**Post date:** [August 21, 2011, 8:26pm UTC](https://discuss.elastic.co/t/impact-of-using-s3-gateway/5213/1 "2011-08-21T20:26:34Z")

</div>

I've been writing the ES/EC2 tutorial that Clinton put me up to. 🙂

I would like to include some content about the downsides of using the S3  
Gateway.

I suppose performance isn't an issue since ES will write asynchronously. I  
know the performance of a recovery is slower because of the EC2/S3 IO  
throughput.

As I see it, there must be some kind of potential consistency problem since  
ES writes to the S3 gateway asynchronously. If the asynchronous write fails  
because of an abrupt termination of a cluster node, isn't is possible that  
the gateway will be in an inconsistent state? I suppose that some other node  
will be the destination of the failed primary shard and could potentially  
correct the problem. If the primary and replica nodes for a shard all fail,  
couldn't the gateway get out of sync?

If yes, can the gateway be corrupted, or will it just recover to a prior  
valid state?

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 22, 2011, 6:39pm UTC](https://discuss.elastic.co/t/impact-of-using-s3-gateway/5213/2 "2011-08-22T18:39:20Z")

</div>

Hi James,

I just saw the tutorial you put up, I will go over it today and update it  
where relevant, but, lemme answer quickly here as well.

The first major downside of using s3 gateway is the overhead in network  
for transferring the (delta) index files and persisting them on s3. The  
second downside is the fact that its written asynchronously, so changes you  
made that have not yet been persisted, will be lost on full cluster shutdown  
(but, the last "fully" persisted state is always available).

-shay.banon

On Sun, Aug 21, 2011 at 11:26 PM, James Cook [jcook@tracermedia.com](mailto:jcook@tracermedia.com) wrote:

> I've been writing the ES/EC2 tutorial that Clinton put me up to. 🙂
> 
> I would like to include some content about the downsides of using the S3  
> Gateway.
> 
> I suppose performance isn't an issue since ES will write asynchronously. I  
> know the performance of a recovery is slower because of the EC2/S3 IO  
> throughput.
> 
> As I see it, there must be some kind of potential consistency problem since  
> ES writes to the S3 gateway asynchronously. If the asynchronous write fails  
> because of an abrupt termination of a cluster node, isn't is possible that  
> the gateway will be in an inconsistent state? I suppose that some other node  
> will be the destination of the failed primary shard and could potentially  
> correct the problem. If the primary and replica nodes for a shard all fail,  
> couldn't the gateway get out of sync?
> 
> If yes, can the gateway be corrupted, or will it just recover to a prior  
> valid state?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:56am UTC](https://discuss.elastic.co/t/impact-of-using-s3-gateway/5213/3 "2017-07-06T03:56:24Z")

</div>

Closed on July 6, 2017, 3:56am UTC