Timeout when creating S3 repository (but success)

Using elasticsearch (with basic licence) v7.7.0, on kubernetes (3 master / 2 data nodes).

Other side, we have a minio server (S3 like).

When creating a snapshot repository:

PUT _snapshot/unsecure_minio
{
  "type": "s3",
  "settings": {
    "bucket": "my-bucket",
    "client": "unsecure"
  }
}

This works, but take more than 1minute with some timeout exception, TRACE logs:

Full https://gist.github.com/ebuildy/322a39b4165eae39bafd2318e24c1a29

2020-06-05T07:18:47.824797878Z "stacktrace": ["java.net.SocketTimeoutException: Read timed out",
2020-06-05T07:18:47.824804767Z "at sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:283) ~[?:?]",
2020-06-05T07:18:47.824809076Z "at sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:309) ~[?:?]",
2020-06-05T07:18:47.824813494Z "at sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350) ~[?:?]",
2020-06-05T07:18:47.82481757Z "at sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803) ~[?:?]",
2020-06-05T07:18:47.824821924Z "at java.net.Socket$SocketInputStream.read(Socket.java:982) ~[?:?]",
2020-06-05T07:18:47.824825859Z "at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[httpcore-4.4.12.jar:4.4.12]",
2020-06-05T07:18:47.824829947Z "at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[httpcore-4.4.12.jar:4.4.12]",
2020-06-05T07:18:47.824833782Z "at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) ~[httpcore-4.4.12.jar:4.4.12]",
2020-06-05T07:18:47.824837819Z "at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[httpclient-4.5.10.jar:4.5.10]",
2020-06-05T07:18:47.824841672Z "at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.10.jar:4.5.10]",
2020-06-05T07:18:47.824845673Z "at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.12.jar:4.4.12]",
2020-06-05T07:18:47.824849545Z "at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.12.jar:4.4.12]",
2020-06-05T07:18:47.824853785Z "at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) ~[httpclient-4.5.10.jar:4.5.10]",
2020-06-05T07:18:47.824857804Z "at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.12.jar:4.4.12]",
2020-06-05T07:18:47.82486293Z "at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:82) ~[aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.824867167Z "at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.12.jar:4.4.12]",
2020-06-05T07:18:47.824871355Z "at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.10.jar:4.5.10]",
2020-06-05T07:18:47.824875153Z "at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.10.jar:4.5.10]",
2020-06-05T07:18:47.824879092Z "at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.10.jar:4.5.10]",
2020-06-05T07:18:47.824896202Z "at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.10.jar:4.5.10]",
2020-06-05T07:18:47.824900233Z "at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.10.jar:4.5.10]",
2020-06-05T07:18:47.824903757Z "at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.824907127Z "at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323) [aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.824910733Z "at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139) [aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.824914112Z "at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796) [aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.824917624Z "at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) [aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.824921141Z "at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) [aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.824924646Z "at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) [aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.824928177Z "at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) [aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.824931717Z "at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) [aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.82493503Z "at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) [aws-java-sdk-core-1.11.749.jar:?]",
2020-06-05T07:18:47.824939113Z "at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) [aws-java-sdk-s3-1.11.749.jar:?]",
2020-06-05T07:18:47.824942492Z "at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000) [aws-java-sdk-s3-1.11.749.jar:?]",
2020-06-05T07:18:47.824945953Z "at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1486) [aws-java-sdk-s3-1.11.749.jar:?]",
2020-06-05T07:18:47.824949439Z "at org.elasticsearch.repositories.s3.S3RetryingInputStream.lambda$openStream$0(S3RetryingInputStream.java:73) [repository-s3-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.824952876Z "at java.security.AccessController.doPrivileged(AccessController.java:312) [?:?]",
2020-06-05T07:18:47.824956287Z "at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:42) [repository-s3-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.824959638Z "at org.elasticsearch.repositories.s3.S3RetryingInputStream.openStream(S3RetryingInputStream.java:73) [repository-s3-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.824963228Z "at org.elasticsearch.repositories.s3.S3RetryingInputStream.<init>(S3RetryingInputStream.java:64) [repository-s3-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.824967027Z "at org.elasticsearch.repositories.s3.S3BlobContainer.readBlob(S3BlobContainer.java:86) [repository-s3-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.824974642Z "at org.elasticsearch.repositories.blobstore.BlobStoreRepository.verify(BlobStoreRepository.java:1896) [elasticsearch-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.824978189Z "at org.elasticsearch.repositories.VerifyNodeRepositoryAction.doVerify(VerifyNodeRepositoryAction.java:126) [elasticsearch-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.824981594Z "at org.elasticsearch.repositories.VerifyNodeRepositoryAction.verify(VerifyNodeRepositoryAction.java:85) [elasticsearch-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.824985035Z "at org.elasticsearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:249) [elasticsearch-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.824988368Z "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.8249918Z "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.7.0.jar:7.7.0]",
2020-06-05T07:18:47.824995267Z "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]",
2020-06-05T07:18:47.824999044Z "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]",
2020-06-05T07:18:47.825002443Z "at java.lang.Thread.run(Thread.java:832) [?:?]"] }

Ok found it, I managed to reproduce it with AWS Java SDK.

The problem was our infrastructure guy, he have added a Nginx in front of Minio, with:

proxy_set_header Expect $http_expect;

With Nginx and Minio, this "stop" the connection after an upload!

Since AWS SDK use the same HTTP connection to upload then read the file, read timeout!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.