Timeout Exception Restoring from s3 Snapshot

Elastic Stack Elasticsearch
1

When I try to restore a snapshot into my elasticsearch cluster, I receive this error after exactly 3min22seconds:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "amazon_client_exception",
        "reason" : "amazon_client_exception: Unable to execute HTTP request: connect timed out"
      }
    ],
    "type" : "amazon_client_exception",
    "reason" : "amazon_client_exception: Unable to execute HTTP request: connect timed out",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "connect timed out"
    }
  },
  "status" : 500
}

I'm not sure how to tell if this is an elasticsearch error, or an issue on the amazon s3 end of things...

2

Elasticsearch Logs Part 1:

[2017-06-27T20:30:37,146][WARN ][r.suppressed             ] path: /_snapshot/s3_repository/snapshot_2017.05.25/_restore, params: {repository=s3_repository, snapshot=snapshot_2017.05.25}
org.elasticsearch.transport.RemoteTransportException: [][:9300][cluster:admin/snapshot/restore]
Caused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: amazon_client_exception: Unable to execute HTTP request: connect timed out
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:618) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:376) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:338) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:287) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3654) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3607) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:600) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:578) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:133) ~[?:?]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:922) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:900) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.getRepositoryData(BlobStoreRepository.java:753) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.snapshots.RestoreService.restoreSnapshot(RestoreService.java:196) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.admin.cluster.snapshots.restore.TransportRestoreSnapshotAction.masterOperation(TransportRestoreSnapshotAction.java:89) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.admin.cluster.snapshots.restore.TransportRestoreSnapshotAction.masterOperation(TransportRestoreSnapshotAction.java:49) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:87) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:166) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.4.0.jar:5.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_25]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_25]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_25]
3

Elasticsearch Logs Part 2 (all part of the same stack trace but it would not fit):

Caused by: java.io.IOException: connect timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_25]
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345) ~[?:1.8.0_25]
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_25]
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_25]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_25]
        at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_25]
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:649) ~[?:?]
        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:542) ~[?:?]
        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:412) ~[?:?]
        at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:134) ~[?:?]
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:179) ~[?:?]
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:328) ~[?:?]
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:612) ~[?:?]
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:447) ~[?:?]
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884) ~[?:?]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[?:?]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:837) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:607) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:376) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:338) ~[?:?]
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:287) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3654) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3607) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:600) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:578) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:133) ~[?:?]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:922) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:900) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.getRepositoryData(BlobStoreRepository.java:753) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.snapshots.RestoreService.restoreSnapshot(RestoreService.java:196) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.admin.cluster.snapshots.restore.TransportRestoreSnapshotAction.masterOperation(TransportRestoreSnapshotAction.java:89) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.admin.cluster.snapshots.restore.TransportRestoreSnapshotAction.masterOperation(TransportRestoreSnapshotAction.java:49) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:87) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:166) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.4.0.jar:5.4.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_25]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_25]
        at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_25]
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.