We have been facing a strange issue while taking snapshot of all the indices in our cluster.
Issuing an API call to take snapshot works fine every time for some indices but fail every time for a few indices. Taking snapshot of the same index separately also fails. On kibana it shows the following error:
INTERNAL_SERVER_ERROR: IndexShardSnapshotFailedException[Failed to perform snapshot (index files)]; nested: IOException[Unable to upload object [rms-profile/indices/<some-id>/0/<some-another-id>] using multipart upload]; nested: SdkClientException[Unable to execute HTTP request: Broken pipe]; nested: SocketException[Broken pipe];
Going through the logs on node, found the logs below:
failed to snapshot shard
org.elasticsearch.index.snapshots.IndexShardSnapshotFailedException: Failed to perform snapshot (index files)
at org.elasticsearch.repositories.blobstore.BlobStoreRepository$2.doRun(BlobStoreRepository.java:1217) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.2.jar:7.5.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: java.io.IOException: Unable to upload object [rms-profile/indices/Ra3oVgiOR2OZsGGw5DdHfg/0/__hG_mX0lWR2aoRTDxJu4-4g] using multipart upload
at org.elasticsearch.repositories.s3.S3BlobContainer.executeMultipartUpload(S3BlobContainer.java:416) ~[?:?]
at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$0(S3BlobContainer.java:99) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:48) ~[?:?]
at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:95) ~[?:?]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotFile(BlobStoreRepository.java:1447) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.access$000(BlobStoreRepository.java:159) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository$2.doRun(BlobStoreRepository.java:1204) ~[elasticsearch-7.5.2.jar:7.5.2]
... 5 more
Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: Broken pipe
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1175) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1121) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668) ~[?:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) ~[?:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4926) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4872) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.doUploadPart(AmazonS3Client.java:3720) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:3705) ~[?:?]
at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeMultipartUpload$20(S3BlobContainer.java:401) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:42) ~[?:?]
at org.elasticsearch.repositories.s3.S3BlobContainer.executeMultipartUpload(S3BlobContainer.java:401) ~[?:?]
at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$0(S3BlobContainer.java:99) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:48) ~[?:?]
at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:95) ~[?:?]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotFile(BlobStoreRepository.java:1447) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.access$000(BlobStoreRepository.java:159) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository$2.doRun(BlobStoreRepository.java:1204) ~[elasticsearch-7.5.2.jar:7.5.2]
... 5 more
Caused by: java.net.SocketException: Broken pipe
at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:421) ~[?:?]
at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:441) ~[?:?]
at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:825) ~[?:?]
at java.net.Socket$SocketOutputStream.write(Socket.java:1007) ~[?:?]
at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124) ~[?:?]
at org.apache.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:136) ~[?:?]
at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:167) ~[?:?]
at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113) ~[?:?]
at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:144) ~[?:?]
at com.amazonaws.http.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:160) ~[?:?]
at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) ~[?:?]
at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) ~[?:?]
at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) ~[?:?]
at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:63) ~[?:?]
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) ~[?:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[?:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[?:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[?:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[?:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[?:?]
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1297) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668) ~[?:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) ~[?:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4926) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4872) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.doUploadPart(AmazonS3Client.java:3720) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.uploadPart(AmazonS3Client.java:3705) ~[?:?]
at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeMultipartUpload$20(S3BlobContainer.java:401) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:42) ~[?:?]
at org.elasticsearch.repositories.s3.S3BlobContainer.executeMultipartUpload(S3BlobContainer.java:401) ~[?:?]
at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$0(S3BlobContainer.java:99) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:48) ~[?:?]
at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:95) ~[?:?]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotFile(BlobStoreRepository.java:1447) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.access$000(BlobStoreRepository.java:159) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository$2.doRun(BlobStoreRepository.java:1204) ~[elasticsearch-7.5.2.jar:7.5.2]
... 5 more
Could not find much help on internet for this. Please provide some directions on what should I further look on to debug and resolve this.