# No progress in index restore
**URL:** https://discuss.elastic.co/t/no-progress-in-index-restore/36035
**Category:** Elasticsearch
**Created:** [December 1, 2015, 11:35am UTC](https://discuss.elastic.co/t/no-progress-in-index-restore/36035 "2015-12-01T11:35:57Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Anatoly\_Petkevich](https://discuss.elastic.co/u/Anatoly_Petkevich)
#### Post date: [December 1, 2015, 11:35am UTC](https://discuss.elastic.co/t/no-progress-in-index-restore/36035/1 "2015-12-01T11:35:58Z")
We have a snapshot of 1TB index on S3 and need to restore in on another cluster.
The version of Elasticsearch is 1.7.2, version of AWS Cloud Plugin is 2.7.1, and number of primary shards is 19.
For the day no shard has been restored and tracking of the restored index via \_status API shows up that size\_in\_bytes property doesn't have a steady grow.
Log file contains a lot of warnings:
[2015-12-01 09:16:29,779][WARN][indices.cluster] [i-51a3e5ef] [[ii-documents][7]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [ii-documents][7] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: [ii-documents][7] restore failed
at org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.restore(IndexShardSnapshotAndRestoreService.java:135)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:109)
... 3 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: [ii-documents][7] failed to restore snapshot [snapshot\_221120150700]
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.restore(BlobStoreIndexShardRepository.java:164)
at org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.restore(IndexShardSnapshotAndRestoreService.java:126)
... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: [ii-documents][7] Failed to recover index
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$RestoreContext.restore(BlobStoreIndexShardRepository.java:780)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.restore(BlobStoreIndexShardRepository.java:162)
... 5 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554)
at sun.security.ssl.InputRecord.read(InputRecord.java:509)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:946)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:903)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:198)
at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178)
at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:151)
at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:151)
at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
at org.elasticsearch.index.snapshots.blobstore.SlicedInputStream.read(SlicedInputStream.java:92)
at java.io.InputStream.read(InputStream.java:101)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$RestoreContext.restoreFile(BlobStoreIndexShardRepository.java:813)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$RestoreContext.restore(BlobStoreIndexShardRepository.java:777)
... 6 more
There is an open issue [https://github.com/elastic/elasticsearch-cloud-aws/issues/149](https://github.com/elastic/elasticsearch-cloud-aws/issues/149), and a related topic [Snapshot restore process is not finished](https://discuss.elastic.co/t/snapshot-restore-process-is-not-finished/21152).
Please advise if there is some solution or workaround of this issue
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [December 1, 2015, 11:54am UTC](https://discuss.elastic.co/t/no-progress-in-index-restore/36035/2 "2015-12-01T11:54:06Z")
Is this happening once or every time you try to restore?
It sounds like here that a Timeout happened when reading S3 buckets.
---
### Author:  [@Anatoly\_Petkevich](https://discuss.elastic.co/u/Anatoly_Petkevich)
#### Post date: [December 1, 2015, 12:13pm UTC](https://discuss.elastic.co/t/no-progress-in-index-restore/36035/3 "2015-12-01T12:13:13Z")
It happens on a regular basis, so that no more than 10% of index data has been restored so far.
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [December 1, 2015, 3:23pm UTC](https://discuss.elastic.co/t/no-progress-in-index-restore/36035/4 "2015-12-01T15:23:33Z")
I added a comment on the issue: [repeated "Read timed out" errors when recovering a large sized shards from S3 repository · Issue #149 · elastic/elasticsearch-cloud-aws · GitHub](https://github.com/elastic/elasticsearch-cloud-aws/issues/149#issuecomment-160998554)
And may be change the default timeout which is `50s` by default.
I'm unsure if this will change anything.
I wonder if the connection is good enough between your machines and S3 buckets. I assume they are in the same region?
The stacktrace shows a typical AWS connection problem. May be we should add a retry by setting [ClientConfiguration (AWS SDK for Java - 1.12.607)](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/ClientConfiguration.html#setMaxErrorRetry%28int%29) but documentation says:
> Sets the maximum number of retry attempts for failed retryable requests (ex: 5xx error responses from services).
I'm unsure if a `SocketTimeoutException` is a retryable request...
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 11:34pm UTC](https://discuss.elastic.co/t/no-progress-in-index-restore/36035/5 "2017-07-05T23:34:26Z")