# S3 gateway snapshot shenanigans **URL:** \ **Category:** Elasticsearch\ **Created:** [January 10, 2011, 7:49pm UTC](https://discuss.elastic.co/t/s3-gateway-snapshot-shenanigans/3760 "2011-01-10T19:49:57Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Jacob\_Perkins](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jacob_perkins/32/3370_2.png) [@Jacob\_Perkins](https://discuss.elastic.co/u/Jacob_Perkins)\ **Post date:** [January 10, 2011, 7:49pm UTC](https://discuss.elastic.co/t/s3-gateway-snapshot-shenanigans/3760/1 "2011-01-10T19:49:57Z")
I've got a very large index ~450gb that I'm in the process of snapshotting to the s3 gateway. After issuing the snapshot command, with the rest api, there's tons of network traffic as shards are copied to s3 until the following: {"ok":true,"\_shards":{"total":16,"successful":12,"failed":4,"failures":[{"index":"tweet-2009q4","shard":10,"reason":"BroadcastShardOperationFailedException[[tweet-2009q4]10] ]; nested: RemoteTransportException[[Corsair][inet[/10.117.110.126:9300]][indices/gateway/snapshot/shard]]; nested: IndexShardGatewaySnapshotFailedException[[tweet-2009q4]10] Failed to perform snapshot (index files)]; nested: AmazonS3Exception[The difference between the request time and the current time is too large.]; "},{"index":"tweet-2009q4","shard":8,"reason":"BroadcastShardOperationFailedException[[tweet-2009q4]8] ]; nested: RemoteTransportException[[Blackheart][inet[/10.117.31.211:9300]][indices/gateway/snapshot/shard]]; nested: IndexShardGatewaySnapshotFailedException[[tweet-2009q4]8] Failed to perform snapshot (index files)]; nested: AmazonS3Exception[The difference between the request time and the current time is too large.]; "},{"index":"tweet-2009q4","shard":9,"reason":"BroadcastShardOperationFailedException[[tweet-2009q4]9] ]; nested: RemoteTransportException[[Silverclaw][inet[/10.102.41.43:9300]][indices/gateway/snapshot/shard]]; nested: IndexShardGatewaySnapshotFailedException[[tweet-2009q4]9] Failed to perform snapshot (index files)]; nested: AmazonS3Exception[The difference between the request time and the current time is too large.]; "},{"index":"tweet-2009q4","shard":14,"reason":"BroadcastShardOperationFailedException[[tweet-2009q4]14] ]; nested: RemoteTransportException[[Noble, Peter][inet[/10.218.55.169:9300]][indices/gateway/snapshot/shard]]; nested: IndexShardGatewaySnapshotFailedException[[tweet-2009q4]14] Failed to perform snapshot (index files)]; nested: AmazonS3Exception[The difference between the request time and the current time is too large.]; "}]}} now, this has happened a few times. First it was 7 failed shards, then 6, then 5, and now 4 as I rerun the same command. Any ideas as to what might be causing this? --jacob ---
**Author:** ![mrflip](https://avatars.discourse-cdn.com/v4/letter/m/a9adbd/32.png) [@mrflip](https://discuss.elastic.co/u/mrflip)\ **Post date:** [January 11, 2011, 1:18am UTC](https://discuss.elastic.co/t/s3-gateway-snapshot-shenanigans/3760/2 "2011-01-11T01:18:55Z")
The error given happens when the request's timestamp is 15 minutes earlier than the request is received. Here's what I think is happening: We start the snapshot at 23:17 ``` [23:17:42,020][DEBUG][index.gateway] ``` [Blackheart] [tweet-2009q4][8] snapshot (api) to s3://infochimps- search/[hoolock][indices][tweet-2009q4][8] ... ES builds out many dozen requests, enough that it fills the pool of available connections: ``` [23:17:43,350][DEBUG] ``` [com.amazonaws.services.s3.internal.S3Signer] Calculated string to sign:"PUT ``` application/octet-stream Mon, 10 Jan 2011 23:17:43 GMT /infochimps-search/hoolock%2Findices ``` %2Ftweet-2009q4%2F8%2F\_\_2o.part72" [23:17:43,350][DEBUG] [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] HttpConnectionManager.getConnection: config = HostConfiguration[host=[http://infochimps-search.s3.amazonaws.com](http://infochimps-search.s3.amazonaws.com)], timeout = 0 [23:17:43,350][DEBUG] [org.apache.commons.httpclient.params.DefaultHttpParams] Set parameter http.protocol.expect-continue = true [23:17:43,350][DEBUG][com.amazonaws.http.HttpClient] Not overwriting Content-Type; already set to: Content-Type: application/ octet-stream [23:17:43,350][DEBUG] [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] Unable to get a connection, waiting..., hostConfig=HostConfiguration[host=http://infochimps- [search.s3.amazonaws.com](http://search.s3.amazonaws.com)] All the connections ahead of it start running; there are enough that some connections don't start within 15 minutes. Right at the 15 minute mark, the connections being popped start failing. Here's the last one that worked and the first of a dozen or so that failed: ``` [23:32:46,379][DEBUG] ``` [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] Getting free connection, hostConfig=HostConfiguration[host=http:// [infochimps-search.s3.amazonaws.com](http://infochimps-search.s3.amazonaws.com)] [23:32:46,387][DEBUG] [org.apache.commons.httpclient.HttpMethodBase] Adding Host request header [23:32:46,393][DEBUG] [org.apache.commons.httpclient.HttpMethodBase] Should close connection in response to directive: close [23:32:46,393][DEBUG] [org.apache.commons.httpclient.HttpConnection] Releasing connection back to connection manager. [23:32:46,393][DEBUG] [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] Freeing connection, hostConfig=HostConfiguration[host=http:// [infochimps-search.s3.amazonaws.com](http://infochimps-search.s3.amazonaws.com)] [23:32:46,393][DEBUG] [org.apache.commons.httpclient.util.IdleConnectionHandler] Adding connection at: 1294702366393 [23:32:46,393][DEBUG] [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] Notifying thread waiting on host pool, hostConfig=HostConfiguration[host=http://infochimps- [search.s3.amazonaws.com](http://search.s3.amazonaws.com)] [23:32:46,394][DEBUG] [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] Getting free connection, hostConfig=HostConfiguration[host=http:// [infochimps-search.s3.amazonaws.com](http://infochimps-search.s3.amazonaws.com)] [23:32:46,394][DEBUG] [org.apache.commons.httpclient.HttpConnection] Open connection to [infochimps-search.s3.amazonaws.com:80](http://infochimps-search.s3.amazonaws.com:80) [23:32:46,398][INFO][com.amazonaws.request] Received error response: Status Code: 403, AWS Request ID: XXXXXX, AWS Error Code: RequestTimeTooSkewed, AWS Error Message: The difference between the request time and the current time is too large., S3 Extended Request ID: XXXXXX =================================================== Takeaways: == 90 percent of the files (taking about 15 minutes to upload) succeeded, yet ES tried to snapshot all of them. If ES compared the MD5 hash of the file in the bucket (an inexpensive AWS API call) with the shard segment it was about to send this can be avoided. == Have the HTTP connections calculate their signature only when the connection becomes available == This is so close to working (succeeded on 12 of 16 shards) that I'm going to try again at 3am and see if it can run 10% faster ๐Ÿ™‚ Apart from that, is there some figure I can munge to increase the size of the HTTP connection pool so that they all start simultaneously? flip On Jan 10, 1:50 pm, jacob [jacob.a.perk...@gmail.com](mailto:jacob.a.perk...@gmail.com) wrote: > I've got a very large index ~450gb that I'm in the process of > snapshotting to the s3 gateway. After issuing the snapshot command, with > the rest api, there's tons of network traffic as shards are copied to s3 > until the following: > > {"ok":true,"\_shards":{"total":16,"successful":12,"failed":4,"failures":[{"i ndex":"tweet-2009q4","shard":10,"reason":"BroadcastShardOperationFailedExce ption[[tweet-2009q4]10] ]; nested: RemoteTransportException[[Corsair][inet[/10.117.110.126:9300]][indices/gate way/snapshot/shard]]; nested: IndexShardGatewaySnapshotFailedException[[tweet-2009q4]10] Failed to perform snapshot (index files)]; nested: AmazonS3Exception[The difference between the request time and the current time is too large.]; "},{"index":"tweet-2009q4","shard":8,"reason":"BroadcastShardOperationFaile dException[[tweet-2009q4]8] ]; nested: RemoteTransportException[[Blackheart][inet[/10.117.31.211:9300]][indices/ga teway/snapshot/shard]]; nested: IndexShardGatewaySnapshotFailedException[[tweet-2009q4]8] Failed to perform snapshot (index files)]; nested: AmazonS3Exception[The difference between the request time and the current time is too large.]; "},{"index":"tweet-2009q4","shard":9,"reason":"BroadcastShardOperationFaile dException[[tweet-2009q4]9] ]; nested: RemoteTransportException[[Silverclaw][inet[/10.102.41.43:9300]][indices/gat eway/snapshot/shard]]; nested: IndexShardGatewaySnapshotFailedException[[tweet-2009q4]9] Failed to perform snapshot (index files)]; nested: AmazonS3Exception[The difference between the request time and the current time is too large.]; "},{"index":"tweet-2009q4","shard":14,"reason":"BroadcastShardOperationFail edException[[tweet-2009q4]14] ]; nested: RemoteTransportException[[Noble, Peter][inet[/10.218.55.169:9300]][indices/gateway/snapshot/shard]]; nested: IndexShardGatewaySnapshotFailedException[[tweet-2009q4]14] Failed to perform snapshot (index files)]; nested: AmazonS3Exception[The difference between the request time and the current time is too large.]; "}]}} > > now, this has happened a few times. First it was 7 failed shards, then > 6, then 5, and now 4 as I rerun the same command. Any ideas as to what > might be causing this? > > --jacob ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 11, 2011, 7:43pm UTC](https://discuss.elastic.co/t/s3-gateway-snapshot-shenanigans/3760/3 "2011-01-11T19:43:59Z")
ES will do the logic of not snapshotting files that are not needed to happen. The problem now is that for the streams it wants to snapshot, they are not bounded, so if it ends up creating 50 files on s3, all are uploaded at one. This should be changed to do them in smaller batches (similar to the way peer recovery works). On Tuesday, January 11, 2011 at 3:18 AM, mrflip wrote: > The error given happens when the request's timestamp is 15 minutes > earlier than the request is received. > > Here's what I think is happening: > > We start the snapshot at 23:17 > > [23:17:42,020][DEBUG][index.gateway] > [Blackheart] [tweet-2009q4][8] snapshot (api) to s3://infochimps- > search/[hoolock][indices][tweet-2009q4][8] ... > > ES builds out many dozen requests, enough that it fills the pool of > available connections: > > [23:17:43,350][DEBUG] > [com.amazonaws.services.s3.internal.S3Signer] Calculated string to > sign:"PUT > > application/octet-stream > Mon, 10 Jan 2011 23:17:43 GMT > /infochimps-search/hoolock%2Findices > %2Ftweet-2009q4%2F8%2F\_\_2o.part72" > [23:17:43,350][DEBUG] > [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] > HttpConnectionManager.getConnection: config = > HostConfiguration[host=[http://infochimps-search.s3.amazonaws.com](http://infochimps-search.s3.amazonaws.com)], > timeout = 0 > [23:17:43,350][DEBUG] > [org.apache.commons.httpclient.params.DefaultHttpParams] Set parameter > http.protocol.expect-continue = true > [23:17:43,350][DEBUG][com.amazonaws.http.HttpClient] Not > overwriting Content-Type; already set to: Content-Type: application/ > octet-stream > [23:17:43,350][DEBUG] > [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] > Unable to get a connection, waiting..., > hostConfig=HostConfiguration[host=http://infochimps- > [search.s3.amazonaws.com](http://search.s3.amazonaws.com)] > > All the connections ahead of it start running; there are enough that > some connections don't start within 15 minutes. > > Right at the 15 minute mark, the connections being popped start > failing. Here's the last one that worked and the first of a dozen or > so that failed: > > [23:32:46,379][DEBUG] > [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] > Getting free connection, hostConfig=HostConfiguration[host=http:// > [infochimps-search.s3.amazonaws.com](http://infochimps-search.s3.amazonaws.com)] > [23:32:46,387][DEBUG] > [org.apache.commons.httpclient.HttpMethodBase] Adding Host request > header > [23:32:46,393][DEBUG] > [org.apache.commons.httpclient.HttpMethodBase] Should close connection > in response to directive: close > [23:32:46,393][DEBUG] > [org.apache.commons.httpclient.HttpConnection] Releasing connection > back to connection manager. > [23:32:46,393][DEBUG] > [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] > Freeing connection, hostConfig=HostConfiguration[host=http:// > [infochimps-search.s3.amazonaws.com](http://infochimps-search.s3.amazonaws.com)] > [23:32:46,393][DEBUG] > [org.apache.commons.httpclient.util.IdleConnectionHandler] Adding > connection at: 1294702366393 > [23:32:46,393][DEBUG] > [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] > Notifying thread waiting on host pool, > hostConfig=HostConfiguration[host=http://infochimps- > [search.s3.amazonaws.com](http://search.s3.amazonaws.com)] > [23:32:46,394][DEBUG] > [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] > Getting free connection, hostConfig=HostConfiguration[host=http:// > [infochimps-search.s3.amazonaws.com](http://infochimps-search.s3.amazonaws.com)] > [23:32:46,394][DEBUG] > [org.apache.commons.httpclient.HttpConnection] Open connection to > [infochimps-search.s3.amazonaws.com:80](http://infochimps-search.s3.amazonaws.com:80) > [23:32:46,398][INFO][com.amazonaws.request] Received > error response: Status Code: 403, AWS Request ID: XXXXXX, AWS Error > Code: RequestTimeTooSkewed, AWS Error Message: The difference between > the request time and the current time is too large., S3 Extended > Request ID: XXXXXX > > =================================================== > > Takeaways: > > == 90 percent of the files (taking about 15 minutes to upload) > succeeded, yet ES tried to snapshot all of them. If ES compared the > MD5 hash of the file in the bucket (an inexpensive AWS API call) with > the shard segment it was about to send this can be avoided. > > == Have the HTTP connections calculate their signature only when the > connection becomes available > > == This is so close to working (succeeded on 12 of 16 shards) that I'm > going to try again at 3am and see if it can run 10% faster ๐Ÿ™‚ Apart > from that, is there some figure I can munge to increase the size of > the HTTP connection pool so that they all start simultaneously? > > flip > > On Jan 10, 1:50 pm, jacob [jacob.a.perk...@gmail.com](mailto:jacob.a.perk...@gmail.com) wrote: > > > I've got a very large index ~450gb that I'm in the process of > > snapshotting to the s3 gateway. After issuing the snapshot command, with > > the rest api, there's tons of network traffic as shards are copied to s3 > > until the following: > > > > {"ok":true,"\_shards":{"total":16,"successful":12,"failed":4,"failures":[{"i ndex":"tweet-2009q4","shard":10,"reason":"BroadcastShardOperationFailedExce ption[[tweet-2009q4]10] ]; nested: RemoteTransportException[[Corsair][inet[/10.117.110.126:9300]][indices/gate way/snapshot/shard]]; nested: IndexShardGatewaySnapshotFailedException[[tweet-2009q4]10] Failed to perform snapshot (index files)]; nested: AmazonS3Exception[The difference between the request time and the current time is too large.]; "},{"index":"tweet-2009q4","shard":8,"reason":"BroadcastShardOperationFaile dException[[tweet-2009q4]8] ]; nested: RemoteTransportException[[Blackheart][inet[/10.117.31.211:9300]][indices/ga teway/snapshot/shard]]; nested: IndexShardGatewaySnapshotFailedException[[tweet-2009q4]8] Failed to perform snapshot (index files)]; nested: AmazonS3Exception[The difference between the request time and the current time is too large.]; "},{"index":"tweet-2009q4","shard":9,"reason":"BroadcastShardOperatio > > nFaile dException[[tweet-2009q4]9] ]; nested: RemoteTransportException[[Silverclaw][inet[/10.102.41.43:9300]][indices/gat eway/snapshot/shard]]; nested: IndexShardGatewaySnapshotFailedException[[tweet-2009q4]9] Failed to perform snapshot (index files)]; nested: AmazonS3Exception[The difference between the request time and the current time is too large.]; "},{"index":"tweet-2009q4","shard":14,"reason":"BroadcastShardOperationFail edException[[tweet-2009q4]14] ]; nested: RemoteTransportException[[Noble, Peter][inet[/10.218.55.169:9300]][indices/gateway/snapshot/shard]]; nested: IndexShardGatewaySnapshotFailedException[[tweet-2009q4]14] Failed to perform snapshot (index files)]; nested: AmazonS3Exception[The difference between the request time and the current time is too large.]; "}]}} > > > > now, this has happened a few times. First it was 7 failed shards, then > > 6, then 5, and now 4 as I rerun the same command. Any ideas as to what > > might be causing this? > > > > --jacob ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 12, 2011, 7:24pm UTC](https://discuss.elastic.co/t/s3-gateway-snapshot-shenanigans/3760/4 "2011-01-12T19:24:15Z")
Posted it on a different thread by mistake, but pushed support for this: [Shared Gateway: Allow to set the number of concurrent streams doing snapshot operations ยท Issue #621 ยท elastic/elasticsearch ยท GitHub](https://github.com/elasticsearch/elasticsearch/issues/issue/621). On Tuesday, January 11, 2011 at 9:43 PM, Shay Banon wrote: > ES will do the logic of not snapshotting files that are not needed to happen. The problem now is that for the streams it wants to snapshot, they are not bounded, so if it ends up creating 50 files on s3, all are uploaded at one. This should be changed to do them in smaller batches (similar to the way peer recovery works). > On Tuesday, January 11, 2011 at 3:18 AM, mrflip wrote: > > > The error given happens when the request's timestamp is 15 minutes > > earlier than the request is received. > > > > Here's what I think is happening: > > > > We start the snapshot at 23:17 > > > > [23:17:42,020][DEBUG][index.gateway]] > > [Blackheart] [tweet-2009q4][8] snapshot (api) to s3://infochimps- > > search/[hoolock][indices][tweet-2009q4][8] ... > > > > ES builds out many dozen requests, enough that it fills the pool of > > available connections: > > > > [23:17:43,350][DEBUG] > > [com.amazonaws.services.s3.internal.S3Signer] Calculated string to > > sign:"PUT > > > > application/octet-stream > > Mon, 10 Jan 2011 23:17:43 GMT > > /infochimps-search/hooloock%2Findices > > %2Ftweet-2009q4%2F8%2F\_\_2o.part72" > > [[23:17:43,350][DEBUG] > > [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] > > HttpConnectionManager.getConnection: config = > > HostConfiguration[host=[http://infochimps-search.s3.amazonaws.com](http://infochimps-search.s3.amazonaws.com)], > > timeout = 0 > > [23:17:43,350][DEBUG] > > [org.apache.commons.httpclient.params.DefaultHttpParams] Set parameter > > http.protocol.expect-continue = true > > [23:17:43,350][DEBUG][com.amazonaws.http.HttpClient] Not > > overwriting Content-Type; already set to: Content-Type: application/ > > octet-stream > > [[23:17:43,350][DEBUG] > > [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] > > Unable to get a connection, waiting..., > > hostConfig=HostConfiguration[host=http://infochimps- > > [search.s3.amazonaws.com](http://search.s3.amazonaws.com)] > > > > All the connections ahead of it start running; there are enough that > > some connections don't start within 15 minutes. > > > > Right at the 15 minute mark, the connections being popped start > > failing. Here's the last one that worked and the first of a dozen or > > so that failed: > > > > [23:32:46,379][DEBUG] > > [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] > > Getting free connection, hostConfig=HostConfiguration[host=http:// > > [infochimps-search.s3.amazonaws.com](http://infochimps-search.s3.amazonaws.com)] > > [23:32:46,387][DEBUG] > > [org.apache.commons.httpclient.HttpMethodBase] Adding Host request > > header > > [23:32:46,393][DEBUG] > > [org.apache.commons.httpclient.HttpMethodBase] Should close connection > > in response to directive: close > > [23:32:46,393][DEBUG] > > [org.apache.commons.httpclient.HttpConnection] Releasing connection > > back to connection manager. > > [23:32:46,393][DEBUG] > > [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] > > Freeing connection, hostConfig=HostConfiguration[host=http:// > > [infochimps-search.s3.amazonaws.com](http://infochimps-search.s3.amazonaws.com)] > > [23:32:46,393][DEBUG] > > [org.apache.commons.httpclient.util.IdleConnectionHandler] Adding > > connection at: 1294702366393 > > [23:32:46,393][DEBUG] > > [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] > > Notifying thread waiting on host pool, > > hostConfig=HostConfiguration[host=http://infochimps- > > [search.s3.amazonaws.com](http://search.s3.amazonaws.com)] > > [23:32:46,394][DEBUG] > > [org.apache.commons.httpclient.MultiThreadedHttpConnectionManager] > > Getting free connection, hostConfig=HostConfiguration[host=http:// > > [infochimps-search.s3.amazonaws.com](http://infochimps-search.s3.amazonaws.com)] > > [23:32:46,394][DEBUG] > > [org.apache.commons.httpclient.HttpConnection] Open connection to > > [infochimps-search.s3.amazonaws.com:80](http://infochimps-search.s3.amazonaws.com:80) > > [23:32:46,398][INFO][com.amazonaws.request] Received > > error response: Status Code: 403, AWS Request ID: XXXXXX, AWS Error > > Code: RequestTimeTooSkewed, AWS Error Message: The difference between > > the request time and the current time is too large., S3 Extended > > Request ID: XXXXXX > > > > =================================================== > > > > Takeaways: > > > > == 90 percent of the files (taking about 15 minutes to upload) > > succeeded, yet ES tried to snapshot all of them. If ES compared the > > MD5 hash of the file in the bucket (an inexpensive AWS API call) with > > the shard segment it was about to send this can be avoided. > > > > == Have the HTTP connections calculate their signature only when the > > connection becomes available > > > > == This is so close to working (succeeded on 12 of 16 shards) that I'm > > going to try again at 3am and see if it can run 10% faster ๐Ÿ™‚ Apart > > from that, is there some figure I can munge to increase the size of > > the HTTP connection pool so that they all start simultaneously? > > > > flip > > > > On Jan 10, 1:50 pm, jacob [jacob.a.perk...@gmail.com](mailto:jacob.a.perk...@gmail.com) wrote: > > > > > I've got a very large index ~450gb that I'm in the process of > > > snapshotting to the s3 gateway. After issuing the snapshot command, with > > > the rest api, there's tons of network traffic as shards are copied to s3 > > > until the following: > > > > > > {"ok":true,"\_shards":{"total":16,"successful":12,"failed":4,"failures":[{"i ndex":"tweet-2009q4","shard":10,"reason":"BroadcastShardOperationFailedExce ption[[tweet-2009q4]10] ]; nested: RemoteTransportException[[Corsair][inet[/10.117.110.126:9300]][indices/gate way/snapshot/shard]]; nested: IndexShardGatewaySnapshotFailedException[[tweet-2009q4]10] Failed to perform snapshot (index files)]; nested: AmazonS3Exception[The difference between the request time and the current time is too large.]; "},{"index":"tweet-2009q4","shard":8,"reason":"BroadcastShardOperationFaile dException[[tweet-2009q4]8] ]; nested: RemoteTransportException[[Blackheart][inet[/10.117.31.211:9300]][indices/ga teway/snapshot/shard]]; nested: IndexShardGatewaySnapshotFailedException[[tweet-2009q4]8] Failed to perform snapshot (index files)]; nested: AmazonS3Exception[The difference between the request time and the current time is too large.]; "},{"index":"tweet-2009q4","shard":9,"reason":"BroadcastShardOperat > > > ionFaile dException[[tweet-2009q4]9] ]; nested: RemoteTransportException[[Silverclaw][inet[/10.102.41.43:9300]][indices/gat eway/snapshot/shard]]; nested: IndexShardGatewaySnapshotFailedException[[tweet-2009q4]9] Failed to perform snapshot (index files)]; nested: AmazonS3Exception[The difference between the request time and the current time is too large.]; "},{"index":"tweet-2009q4","shard":14,"reason":"BroadcastShardOperationFail edException[[tweet-2009q4]14] ]; nested: RemoteTransportException[[Noble, Peter][inet[/10.218.55.169:9300]][indices/gateway/snapshot/shard]]; nested: IndexShardGatewaySnapshotFailedException[[tweet-2009q4]14] Failed to perform snapshot (index files)]; nested: AmazonS3Exception[The difference between the request time and the current time is too large.]; "}]}} > > > > > > now, this has happened a few times. First it was 7 failed shards, then > > > 6, then 5, and now 4 as I rerun the same command. Any ideas as to what > > > might be causing this? > > > > > > --jacob ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:13am UTC](https://discuss.elastic.co/t/s3-gateway-snapshot-shenanigans/3760/5 "2017-07-06T04:13:58Z")