I am running a 4 node cluster in AWS trying to use the Repository Analysis API to validate S3 compatible storage. The cluster is healthy and well provisioned, and doing nothing but running this API test. After continuous failures I tested with an AWS S3 bucket in the same region and the analysis is failing with the same errors which I will include below (error below analyzing AWS S3). The API runs to completion at smaller scales, but fails when run at the scale recommended in the Repo Analysis API documentation. Any help or advice appreciated.
[2023-02-23T23:09:41,175][WARN ][r.suppressed ] [node-4] path: /_snapshot/aws_repository/_analyze, params: {pretty=true, detailed=true, max_blob_size=2gb, blob_count=2000, max_total_data_size=1t
b, repository=aws_repository, timeout=1440m}
org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] analysis failed, you may need to manually remove [temp-analysis-QGCHbkHJR72lxW-3ghvodA]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-1][10.114.132.215:9300][cluster:admin/repository/analyze/blob]
Suppressed: org.elasticsearch.transport.RemoteTransportException: [node-3][10.114.134.133:9300][cluster:admin/repository/analyze/blob]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] failure processing [blob analysis [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-39-ArHGp_
7WQuaDia_AKSmIsA, length=536870912, seed=4786312329708984008, readEarly=false, writeAndOverwrite=false, abortWrite=false]]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] [GetRepositoryBlobChecksumRequest{retrieve [whole blob] from [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-39-ArHGp_7WQuaDia_AKSmIsA]}] (after write complete) failed on node [{node-2}{ycjlcqMeRiCx7kkbY4JX5w}{b2ggL3TgQPaSqP-68m8nUw}{node-2}{10.114.134.98}{10.114.134.98:9300}{cdfhilmrstw}{ml.allocated_processors_double=16.0, xpack.installed=true, ml.machine_memory=66594594816, ml.allocated_processors=16, ml.max_jvm_size=33285996544}]
Suppressed: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] [GetRepositoryBlobChecksumRequest{retrieve [whole blob] from [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-39-ArHGp_7WQuaDia_AKSmIsA]}] (after write complete) failed on node [{node-1}{lSc7o9KCT6-agbarqS8XpA}{fGTrOYaKQJymFqX4PCTXdA}{node-1}{10.114.132.215}{10.114.132.215:9300}{cdfhilmrstw}{ml.allocated_processors_double=16.0, xpack.installed=true, ml.machine_memory=65889968128, ml.allocated_processors=16, ml.max_jvm_size=32950452224}]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-1][10.114.132.215:9300][cluster:admin/repository/analyze/blob/read]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] cancelled [retrieve [whole blob] from [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-39-ArHGp_7WQuaDia_AKSmIsA]] after reading [33032799] bytes
Suppressed: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] [GetRepositoryBlobChecksumRequest{retrieve [whole blob] from [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-39-ArHGp_7WQuaDia_AKSmIsA]}] (after write complete) failed on node [{node-3}{QIiwF4ejR3afmE6DAW2Kpg}{jPDA492ES1aeSRnoohrmCQ}{node-3}{10.114.134.133}{10.114.134.133:9300}{cdfhilmrstw}{ml.allocated_processors_double=16.0, xpack.installed=true, ml.machine_memory=65889968128, ml.allocated_processors=16, ml.max_jvm_size=32950452224}]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-3][10.114.134.133:9300][cluster:admin/repository/analyze/blob/read]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] cancelled [retrieve [whole blob] from [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-39-ArHGp_7WQuaDia_AKSmIsA]] after reading [34133282] bytes
Suppressed: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] [GetRepositoryBlobChecksumRequest{retrieve [whole blob] from [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-39-ArHGp_7WQuaDia_AKSmIsA]}] (after write complete) failed on node [{node-4}{YGyB0PtrRMW67ceo9PCs0w}{zuBBhmw_SQqsVQBWwGmntQ}{node-4}{10.114.138.177}{10.114.138.177:9300}{cdfhilmrstw}{ml.allocated_processors_double=16.0, xpack.installed=true, ml.machine_memory=65889968128, ml.allocated_processors=16, ml.max_jvm_size=32950452224}]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-4][10.114.138.177:9300][cluster:admin/repository/analyze/blob/read]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] cancelled [retrieve [whole blob] from [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-39-ArHGp_7WQuaDia_AKSmIsA]] after reading [34080256] bytes
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-2][10.114.134.98:9300][cluster:admin/repository/analyze/blob/read]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] cancelled [retrieve [whole blob] from [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-39-ArHGp_7WQuaDia_AKSmIsA]] after reading [33918208] bytes
Suppressed: org.elasticsearch.transport.RemoteTransportException: [node-1][10.114.132.215:9300][cluster:admin/repository/analyze/blob]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] failure processing [blob analysis [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-52-jkcfMnWhTa-qFlVg-vvC_g, length=1073741824, seed=6390949861154951734, readEarly=false, writeAndOverwrite=false, abortWrite=false]]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] blob upload cancelled at position [46432256+4096/1073741824]
Suppressed: org.elasticsearch.transport.RemoteTransportException: [node-1][10.114.132.215:9300][cluster:admin/repository/analyze/blob]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] failure processing [blob analysis [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-25-QRKKmXtWRBqrYWyXsyN2vA, length=2147483648, seed=-8321337411330896189, readEarly=false, writeAndOverwrite=false, abortWrite=false]]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] blob upload cancelled at position [241532928+4096/2147483648]
Suppressed: org.elasticsearch.transport.RemoteTransportException: [node-1][10.114.132.215:9300][cluster:admin/repository/analyze/blob]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] failure processing [blob analysis [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-8-jmi5M2WKQnGAH_5rh4lkSQ, length=536870912, seed=2629957976200822414, readEarly=false, writeAndOverwrite=false, abortWrite=false]]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] blob upload cancelled at position [247925188/536870912]
Suppressed: org.elasticsearch.transport.RemoteTransportException: [node-3][10.114.134.133:9300][cluster:admin/repository/analyze/blob]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] failure processing [blob analysis [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-49-PddSeoA7RauhA87dlW8Emw, length=268435456, seed=1767012607676872037, readEarly=false, writeAndOverwrite=false, abortWrite=false]]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] [GetRepositoryBlobChecksumRequest{retrieve [whole blob] from [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-49-PddSeoA7RauhA87dlW8Emw]}] (after write complete) failed on node [{node-1}{lSc7o9KCT6-agbarqS8XpA}{fGTrOYaKQJymFqX4PCTXdA}{node-1}{10.114.132.215}{10.114.132.215:9300}{cdfhilmrstw}{ml.allocated_processors_double=16.0, xpack.installed=true, ml.machine_memory=65889968128, ml.allocated_processors=16, ml.max_jvm_size=32950452224}]
Suppressed: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] [GetRepositoryBlobChecksumRequest{retrieve [whole blob] from [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-49-PddSeoA7RauhA87dlW8Emw]}] (after write complete) failed on node [{node-4}{YGyB0PtrRMW67ceo9PCs0w}{zuBBhmw_SQqsVQBWwGmntQ}{node-4}{10.114.138.177}{10.114.138.177:9300}{cdfhilmrstw}{ml.allocated_processors_double=16.0, xpack.installed=true, ml.machine_memory=65889968128, ml.allocated_processors=16, ml.max_jvm_size=32950452224}]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-4][10.114.138.177:9300][cluster:admin/repository/analyze/blob/read]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] cancelled [retrieve [whole blob] from [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-49-PddSeoA7RauhA87dlW8Emw]] after reading [2578856] bytes
Suppressed: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] [GetRepositoryBlobChecksumRequest{retrieve [whole blob] from [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-49-PddSeoA7RauhA87dlW8Emw]}] (after write complete) failed on node [{node-2}{ycjlcqMeRiCx7kkbY4JX5w}{b2ggL3TgQPaSqP-68m8nUw}{node-2}{10.114.134.98}{10.114.134.98:9300}{cdfhilmrstw}{ml.allocated_processors_double=16.0, xpack.installed=true, ml.machine_memory=66594594816, ml.allocated_processors=16, ml.max_jvm_size=33285996544}]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-2][10.114.134.98:9300][cluster:admin/repository/analyze/blob/read]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] cancelled [retrieve [whole blob] from [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-49-PddSeoA7RauhA87dlW8Emw]] after reading [3214976] bytes
Suppressed: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] [GetRepositoryBlobChecksumRequest{retrieve [whole blob] from [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-49-PddSeoA7RauhA87dlW8Emw]}] (after write complete) failed on node [{node-3}{QIiwF4ejR3afmE6DAW2Kpg}{jPDA492ES1aeSRnoohrmCQ}{node-3}{10.114.134.133}{10.114.134.133:9300}{cdfhilmrstw}{ml.allocated_processors_double=16.0, xpack.installed=true, ml.machine_memory=65889968128, ml.allocated_processors=16, ml.max_jvm_size=32950452224}]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-3][10.114.134.133:9300][cluster:admin/repository/analyze/blob/read]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] cancelled [retrieve [whole blob] from [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-49-PddSeoA7RauhA87dlW8Emw]] after reading [2984818] bytes
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-1][10.114.132.215:9300][cluster:admin/repository/analyze/blob/read]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] cancelled [retrieve [whole blob] from [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-49-PddSeoA7RauhA87dlW8Emw]] after reading [3005939] bytes
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] failure processing [blob analysis [aws_repository:temp-analysis-QGCHbkHJR72lxW-3ghvodA/test-blob-37-pHCj13S6QL-46rNvry4sag, length=536870912, seed=1328922046822391799, readEarly=false, writeAndOverwrite=false, abortWrite=false]]
Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [aws_repository] blob upload cancelled at position [220370664/536870912]