Snapshot failed due to connection timeout to S3

[feiwhomai]

Hi All,

I am trying to make a snapshot, but it always fails after around 1.5 hours. The exception shows the connection to S3 timeouts.

The cluster version is 6.8.2

It used to work fine, but starting from 2 weeks back, it fails like this. The snapshot status is always PARTIAL. I am not sure if this is because of the datasize?

I also followed https://aws.amazon.com/premiumsupport/knowledge-center/vpc-find-cause-of-failed-dns-queries/ but the number looks good.

Is there anything else I should check? Thanks in advance!

Caused by: java.io.IOException: Unable to upload object [from_20200612_new/indices/42kvMHFNSyat_d6idkixIw/191/__XpFbu1seQKuR-78KwG8qGQ.part0] using multipart upload
	at org.elasticsearch.repositories.s3.S3BlobContainer.executeMultipartUpload(S3BlobContainer.java:285) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$2(S3BlobContainer.java:102) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:551) ~[?:?]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:48) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:98) ~[?:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$SnapshotContext.snapshotFile(BlobStoreRepository.java:1298) ~[elasticsearch-6.8.2.jar:6.8.2]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$SnapshotContext.snapshot(BlobStoreRepository.java:1234) ~[elasticsearch-6.8.2.jar:6.8.2]
	... 9 more
Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to my-es-s3-repository.s3.ap-northeast-2.amazonaws.com:443 [my-es-s3-repository.s3.ap-northeast-2.amazonaws.com/52.219.56.7] failed: connect timed out