# Timeout when creating S3 repository (but success)

**URL:** <https://discuss.elastic.co/t/timeout-when-creating-s3-repository-but-success/235905>\
**Category:** Elasticsearch\
**Created:** [June 5, 2020, 7:44am UTC](https://discuss.elastic.co/t/timeout-when-creating-s3-repository-but-success/235905 "2020-06-05T07:44:34Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\
**Post date:** [June 5, 2020, 7:44am UTC](https://discuss.elastic.co/t/timeout-when-creating-s3-repository-but-success/235905/1 "2020-06-05T07:44:34Z")

</div>

Using elasticsearch (with basic licence) v7.7.0, on kubernetes (3 master / 2 data nodes).

Other side, we have a minio server (S3 like).

When creating a snapshot repository:

```auto
PUT _snapshot/unsecure_minio
{
  "type": "s3",
  "settings": {
    "bucket": "my-bucket",
    "client": "unsecure"
  }
}

```

This works, but take more than 1minute with some timeout exception, TRACE logs:

Full [https://gist.github.com/ebuildy/322a39b4165eae39bafd2318e24c1a29](https://gist.github.com/ebuildy/322a39b4165eae39bafd2318e24c1a29)

```auto
2020-06-05T07:18:47.824797878Z "stacktrace": ["java.net.SocketTimeoutException: Read timed out",
2020-06-05T07:18:47.824804767Z "at sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:283) ~[?:?]",
2020-06-05T07:18:47.824809076Z "at sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:309) ~[?:?]",
2020-06-05T07:18:47.824813494Z "at sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350) ~[?:?]",
2020-06-05T07:18:47.82481757Z "at sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803) ~[?:?]",
2020-06-05T07:18:47.824821924Z "at java.net.Socket$SocketInputStream.read(Socket.java:982) ~[?:?]",
2020-06-05T07:18:47.824825859Z "at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[httpcore-4.4.12.jar:4.4.12]",
2020-06-05T07:18:47.824829947Z "at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[httpcore-4.4.12.jar:4.4.12]",
2020-06-05T07:18:47.824833782Z "at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) ~[httpcore-4.4.12.jar:4.4.12]",
2020-06-05T07:18:47.824837819Z "at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[httpclient-4.5.10.jar:4.5.10]",
2020-06-05T07:18:47.824841672Z "at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.10.jar:4.5.10]",
2020-06-05T07:18:47.824845673Z "at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.12.jar:4.4.12]",
2020-06-05T07:18:47.824849545Z "at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.12.jar:4.4.12]",
2020-06-05T07:18:47.824853785Z "at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) ~[httpclient-4.5.10.jar:4.5.10]",
2020-06-05T07:18:47.824857804Z "at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.12.jar:4.4.12]",
2020-06-05T07:18:47.82486293Z "at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:82) ~[aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.824867167Z "at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.12.jar:4.4.12]",
2020-06-05T07:18:47.824871355Z "at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.10.jar:4.5.10]",
2020-06-05T07:18:47.824875153Z "at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.10.jar:4.5.10]",
2020-06-05T07:18:47.824879092Z "at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.10.jar:4.5.10]",
2020-06-05T07:18:47.824896202Z "at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.10.jar:4.5.10]",
2020-06-05T07:18:47.824900233Z "at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.10.jar:4.5.10]",
2020-06-05T07:18:47.824903757Z "at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.824907127Z "at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323) [aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.824910733Z "at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139) [aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.824914112Z "at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796) [aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.824917624Z "at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) [aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.824921141Z "at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) [aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.824924646Z "at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) [aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.824928177Z "at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) [aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.824931717Z "at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) [aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.82493503Z "at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) [aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.824939113Z "at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) [aws-java-sdk-s3-1.11.749.jar:?]",
2020-06-05T07:18:47.824942492Z "at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000) [aws-java-sdk-s3-1.11.749.jar:?]",
2020-06-05T07:18:47.824945953Z "at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1486) [aws-java-sdk-s3-1.11.749.jar:?]",
2020-06-05T07:18:47.824949439Z "at org.elasticsearch.repositories.s3.S3RetryingInputStream.lambda$openStream$0(S3RetryingInputStream.java:73) [repository-s3-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.824952876Z "at java.security.AccessController.doPrivileged(AccessController.java:312) [?:?]",
2020-06-05T07:18:47.824956287Z "at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:42) [repository-s3-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.824959638Z "at org.elasticsearch.repositories.s3.S3RetryingInputStream.openStream(S3RetryingInputStream.java:73) [repository-s3-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.824963228Z "at org.elasticsearch.repositories.s3.S3RetryingInputStream.<init>(S3RetryingInputStream.java:64) [repository-s3-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.824967027Z "at org.elasticsearch.repositories.s3.S3BlobContainer.readBlob(S3BlobContainer.java:86) [repository-s3-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.824974642Z "at org.elasticsearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:1896) [elasticsearch-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.824978189Z "at org.elasticsearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:126) [elasticsearch-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.824981594Z "at org.elasticsearch.repositories.VerifyNodeRepositoryAction.verify(VerifyNodeRepositoryAction.java:85) [elasticsearch-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.824985035Z "at org.elasticsearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:249) [elasticsearch-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.824988368Z "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.8249918Z "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.824995267Z "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]",
2020-06-05T07:18:47.824999044Z "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]",
2020-06-05T07:18:47.825002443Z "at java.lang.Thread.run(Thread.java:832) [?:?]"] }

```

---

<div class="post-metadata">

**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\
**Post date:** [June 8, 2020, 8:26am UTC](https://discuss.elastic.co/t/timeout-when-creating-s3-repository-but-success/235905/2 "2020-06-08T08:26:54Z")

</div>

Ok found it, I managed to reproduce it with AWS Java SDK.

The problem was our infrastructure guy, he have added a Nginx in front of Minio, with:

```
proxy_set_header Expect $http_expect;

```

With Nginx and Minio, this "stop" the connection after an upload!

Since AWS SDK use the same HTTP connection to upload then read the file, read timeout!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2020, 8:26am UTC](https://discuss.elastic.co/t/timeout-when-creating-s3-repository-but-success/235905/3 "2020-07-06T08:26:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.