# Incremental snapshots are failing in "Trying to write new repository data over unfinished write"

**URL:** <https://discuss.elastic.co/t/incremental-snapshots-are-failing-in-trying-to-write-new-repository-data-over-unfinished-write/372408>\
**Category:** Elasticsearch\
**Tags:** snapshot-and-restore\
**Created:** [December 26, 2024, 11:10am UTC](https://discuss.elastic.co/t/incremental-snapshots-are-failing-in-trying-to-write-new-repository-data-over-unfinished-write/372408 "2024-12-26T11:10:17Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rajesh\_Kannan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rajesh_kannan/32/47154_2.png) [@Rajesh\_Kannan](https://discuss.elastic.co/u/Rajesh_Kannan)\
**Post date:** [December 26, 2024, 11:10am UTC](https://discuss.elastic.co/t/incremental-snapshots-are-failing-in-trying-to-write-new-repository-data-over-unfinished-write/372408/1 "2024-12-26T11:10:18Z")

</div>

The incremental snapshot is failing with the below error. There is only one successful backup created and the daily crons are failing with the below errors.

{"type": "server", "timestamp": "2024-12-26T06:46:46,097Z", "level": "INFO", "component": "o.e.r.b.BlobStoreRepository", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Trying to write new repository data over unfinished write, repo [backup-s3-repository] is at safe generation [0] and pending generation [2]", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,336Z", "level": "DEBUG", "component": "c.a.request", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Sending Request: HEAD [http://myhost.mynw.com](http://myhost.mynw.com) /inp44olio-prdapp1-es/olio-elastic-prod/index-0 Headers: (amz-sdk-invocation-id: f2ae0226-98f8-def1-64a9-9b84a192ac36, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium, ) ", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,336Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:0/0/500\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064646Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,336Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 String to Sign: '"AWS4-HMAC-SHA256\n20241226T064646Z\n20241226/us-east-1/s3/aws4\_request\ne0e301b0381eb953db42bf35894c3ce4dcd3b6d39a4ef1d18f1b169b4b9009c5"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,337Z", "level": "DEBUG", "component": "c.a.request", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retrying Request: HEAD [http://myhost.mynw.com](http://myhost.mynw.com) /inp44olio-prdapp1-es/olio-elastic-prod/index-0 Headers: (amz-sdk-invocation-id: f2ae0226-98f8-def1-64a9-9b84a192ac36, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium, ) ", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,337Z", "level": "DEBUG", "component": "c.a.h.AmazonHttpClient", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retriable error detected, will retry in 14ms, attempt number: 0", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,370Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:1/14/495\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064646Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,370Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 String to Sign: '"AWS4-HMAC-SHA256\n20241226T064646Z\n20241226/us-east-1/s3/aws4\_request\n634de39f28bb52495cf02d05954c4285b1bdb0639777dce1cda01efcdfdd495b"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,371Z", "level": "DEBUG", "component": "c.a.request", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retrying Request: HEAD [http://myhost.mynw.com](http://myhost.mynw.com) /inp44olio-prdapp1-es/olio-elastic-prod/index-0 Headers: (amz-sdk-invocation-id: f2ae0226-98f8-def1-64a9-9b84a192ac36, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium, ) ", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,372Z", "level": "DEBUG", "component": "c.a.h.AmazonHttpClient", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retriable error detected, will retry in 15ms, attempt number: 1", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,387Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:2/15/490\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064646Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,387Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 String to Sign: '"AWS4-HMAC-SHA256\n20241226T064646Z\n20241226/us-east-1/s3/aws4\_request\nc7894cb9600d567a56fa0164e1aec018e1b1918a7669207de47bad05d6ef4b9c"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,388Z", "level": "DEBUG", "component": "c.a.request", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retrying Request: HEAD [http://myhost.mynw.com](http://myhost.mynw.com) /inp44olio-prdapp1-es/olio-elastic-prod/index-0 Headers: (amz-sdk-invocation-id: f2ae0226-98f8-def1-64a9-9b84a192ac36, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium, ) ", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,388Z", "level": "DEBUG", "component": "c.a.h.AmazonHttpClient", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retriable error detected, will retry in 342ms, attempt number: 2", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,730Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:3/342/485\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064646Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,731Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 String to Sign: '"AWS4-HMAC-SHA256\n20241226T064646Z\n20241226/us-east-1/s3/aws4\_request\nfe860cd2099cb03de0c9eccaa1a4d98f93ee11fc347101ba1f601c3c64d06a16"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,732Z", "level": "DEBUG", "component": "c.a.request", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retrying Request: HEAD [http://myhost.mynw.com](http://myhost.mynw.com) /inp44olio-prdapp1-es/olio-elastic-prod/index-0 Headers: (amz-sdk-invocation-id: f2ae0226-98f8-def1-64a9-9b84a192ac36, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium, ) ", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,732Z", "level": "DEBUG", "component": "c.a.h.AmazonHttpClient", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retriable error detected, will retry in 113ms, attempt number: 3", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,845Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:4/113/480\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064646Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,846Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 String to Sign: '"AWS4-HMAC-SHA256\n20241226T064646Z\n20241226/us-east-1/s3/aws4\_request\n63daafe677d4a23c7212b71ed12ef82cbf59ab0d98cb584511194dcd58a390b8"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,856Z", "level": "DEBUG", "component": "c.a.request", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retrying Request: HEAD [http://myhost.mynw.com](http://myhost.mynw.com) /inp44olio-prdapp1-es/olio-elastic-prod/index-0 Headers: (amz-sdk-invocation-id: f2ae0226-98f8-def1-64a9-9b84a192ac36, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium, ) ", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:46,856Z", "level": "DEBUG", "component": "c.a.h.AmazonHttpClient", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retriable error detected, will retry in 659ms, attempt number: 4", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:47,515Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:5/659/475\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064647Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:47,517Z", "level": "DEBUG", "component": "c.a.h.AmazonHttpClient", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retriable error detected, will retry in 1582ms, attempt number: 5", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:49,099Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:6/1582/470\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064649Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:49,099Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 String to Sign: '"AWS4-HMAC-SHA256\n20241226T064649Z\n20241226/us-east-1/s3/aws4\_request\n3967b1a30aa706f881e979c7b3f0a7b7d35a450be452a76dfebd668dbd8cb948"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:49,102Z", "level": "DEBUG", "component": "c.a.request", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retrying Request: HEAD [http://myhost.mynw.com](http://myhost.mynw.com) /inp44olio-prdapp1-es/olio-elastic-prod/index-0 Headers: (amz-sdk-invocation-id: f2ae0226-98f8-def1-64a9-9b84a192ac36, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium, ) ", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:49,102Z", "level": "DEBUG", "component": "c.a.h.AmazonHttpClient", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retriable error detected, will retry in 1262ms, attempt number: 6", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:50,364Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:7/1262/465\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064650Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:50,365Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 String to Sign: '"AWS4-HMAC-SHA256\n20241226T064650Z\n20241226/us-east-1/s3/aws4\_request\nffddd6496d90f82da3e84ea8cc841f58da1b3bca8a54db63f23e7c65aaf752e8"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:50,366Z", "level": "DEBUG", "component": "c.a.request", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retrying Request: HEAD [http://myhost.mynw.com](http://myhost.mynw.com) /inp44olio-prdapp1-es/olio-elastic-prod/index-0 Headers: (amz-sdk-invocation-id: f2ae0226-98f8-def1-64a9-9b84a192ac36, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium, ) ", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:50,366Z", "level": "DEBUG", "component": "c.a.h.AmazonHttpClient", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retriable error detected, will retry in 5619ms, attempt number: 7", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:55,986Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:8/5619/460\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064655Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:55,986Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 String to Sign: '"AWS4-HMAC-SHA256\n20241226T064655Z\n20241226/us-east-1/s3/aws4\_request\n87f579bee63261074c6bbf38f43bc516d63db09df406480ac6ed02efd142d074"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:55,988Z", "level": "DEBUG", "component": "c.a.request", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retrying Request: HEAD [http://myhost.mynw.com](http://myhost.mynw.com) /inp44olio-prdapp1-es/olio-elastic-prod/index-0 Headers: (amz-sdk-invocation-id: f2ae0226-98f8-def1-64a9-9b84a192ac36, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium, ) ", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:46:55,988Z", "level": "DEBUG", "component": "c.a.h.AmazonHttpClient", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retriable error detected, will retry in 19823ms, attempt number: 8", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:47:05,490Z", "level": "INFO", "component": "o.e.x.s.a.AuthorizationService", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Took [70ms] to resolve [4063] indices for action [indices:data/read/search] and user [olio-prod]", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:47:15,811Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:9/19823/455\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064715Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:47:15,811Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 String to Sign: '"AWS4-HMAC-SHA256\n20241226T064715Z\n20241226/us-east-1/s3/aws4\_request\n408e57144f9f163f607670f1ec7e5b834287c62ff443ba68fd7d1b5cc06c2482"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:47:15,813Z", "level": "DEBUG", "component": "c.a.request", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retrying Request: HEAD [http://myhost.mynw.com](http://myhost.mynw.com) /inp44olio-prdapp1-es/olio-elastic-prod/index-0 Headers: (amz-sdk-invocation-id: f2ae0226-98f8-def1-64a9-9b84a192ac36, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium, ) ", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:47:15,813Z", "level": "DEBUG", "component": "c.a.h.AmazonHttpClient", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retriable error detected, will retry in 99ms, attempt number: 9", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:47:15,912Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:10/99/450\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK\_64-Bit\_Server\_VM/17.0.1+12 java/17.0.1 vendor/Eclipse\_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064715Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:47:15,913Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 String to Sign: '"AWS4-HMAC-SHA256\n20241226T064715Z\n20241226/us-east-1/s3/aws4\_request\n405d2ce8360ad3d14231bcfbe04c0c20f355b46899a81c28a55383ea3d14c63f"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }  
{"type": "server", "timestamp": "2024-12-26T06:47:15,915Z", "level": "WARN", "component": "o.e.s.SnapshotsService", "cluster.name": "es", "node.name": "es-es-master-0", "message": "[backup-s3-repository:snapshot-202412260622/0lwk886URJes3jGDS-hrMQ] failed to finalize snapshot", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" ,  
"stacktrace": ["org.elasticsearch.snapshots.SnapshotException: [backup-s3-repository:snapshot-202412260622/0lwk886URJes3jGDS-hrMQ] failed to update snapshot in repository",  
"at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$42(BlobStoreRepository.java:1377) [elasticsearch-7.16.2.jar:7.16.2]",  
"at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:144) [elasticsearch-7.16.2.jar:7.16.2]",  
"at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:144) [elasticsearch-7.16.2.jar:7.16.2]",  
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:138) [elasticsearch-7.16.2.jar:7.16.2]",  
"at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) [elasticsearch-7.16.2.jar:7.16.2]",  
"at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:100) [elasticsearch-7.16.2.jar:7.16.2]",  
"at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:131) [elasticsearch-7.16.2.jar:7.16.2]",  
"at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:139) [elasticsearch-7.16.2.jar:7.16.2]",  
"at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:52) [elasticsearch-7.16.2.jar:7.16.2]",  
"at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:29) [elasticsearch-7.16.2.jar:7.16.2]",  
"at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$63(BlobStoreRepository.java:2300) [elasticsearch-7.16.2.jar:7.16.2]",  
"at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) [elasticsearch-7.16.2.jar:7.16.2]",  
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.16.2.jar:7.16.2]",  
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.16.2.jar:7.16.2]",  
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]",  
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]",  
"at java.lang.Thread.run(Thread.java:833) [?:?]",  
"Caused by: org.elasticsearch.common.blobstore.BlobStoreException: Failed to check if blob [index-0] exists",  
"at org.elasticsearch.repositories.s3.S3BlobContainer.blobExists(S3BlobContainer.java:90) ~[?:?]",  
"at org.elasticsearch.repositories.blobstore.BlobStoreRepository.ensureSafeGenerationExists(BlobStoreRepository.java:2462) ~[elasticsearch-7.16.2.jar:7.16.2]",  
"at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$66(BlobStoreRepository.java:2316) ~[elasticsearch-7.16.2.jar:7.16.2]",  
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.16.2.jar:7.16.2]",  
"... 13 more",  
"Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connection reset",  
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1201) ~[?:?]",  
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1147) ~[?:?]",  
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796) ~[?:?]",  
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) ~[?:?]",  
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) ~[?:?]",  
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) ~[?:?]",  
"at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) ~[?:?]",  
"at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) ~[?:?]",  
"at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) ~[?:?]",  
"at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) ~[?:?]",  
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.16.2.jar:7.16.2]",  
"... 13 more",  
"Caused by: java.net.SocketException: Connection reset",  
"at sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:323) ~[?:?]",  
"at sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350) ~[?:?]",  
"at sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803) ~[?:?]",  
"at java.net.Socket$SocketInputStream.read(Socket.java:966) ~[?:?]",  
"at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[?:?]",  
"at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[?:?]",  
"at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) ~[?:?]",  
"at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[?:?]",  
"at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[?:?]",  
"at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[?:?]",  
"at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[?:?]",  
"at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) ~[?:?]",  
"at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[?:?]",  
"at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:82) ~[?:?]",  
"at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[?:?]",  
"at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[?:?]",  
"at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[?:?]",  
"at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[?:?]",  
"at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[?:?]",  
"at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[?:?]",  
"at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[?:?]",  
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323) ~[?:?]",  
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139) ~[?:?]",  
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796) ~[?:?]",  
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) ~[?:?]",  
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) ~[?:?]",  
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) ~[?:?]",  
"at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) ~[?:?]",  
"at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) ~[?:?]",  
"at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) ~[?:?]",  
"at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) ~[?:?]",  
"at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000) ~[?:?]",  
"at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1335) ~[?:?]",  
"at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1309) ~[?:?]",  
"at com.amazonaws.services.s3.AmazonS3Client.doesObjectExist(AmazonS3Client.java:1390) ~[?:?]",  
"at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$blobExists$0(S3BlobContainer.java:88) ~[?:?]",  
"at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]",  
"at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) ~[?:?]",  
"at org.elasticsearch.repositories.s3.S3BlobContainer.blobExists(S3BlobContainer.java:88) ~[?:?]",  
"at org.elasticsearch.repositories.blobstore.BlobStoreRepository.ensureSafeGenerationExists(BlobStoreRepository.java:2462) ~[elasticsearch-7.16.2.jar:7.16.2]",  
"at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$66(BlobStoreRepository.java:2316) ~[elasticsearch-7.16.2.jar:7.16.2]",  
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.16.2.jar:7.16.2]",

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [December 27, 2024, 8:32am UTC](https://discuss.elastic.co/t/incremental-snapshots-are-failing-in-trying-to-write-new-repository-data-over-unfinished-write/372408/2 "2024-12-27T08:32:31Z")

</div>

> [@Rajesh\_Kannan](#):
>
> ... [elasticsearch-7.16.2.jar:7.16.2]

This version is very old and no longer supported. You need to upgrade (ideally to an 8.x version) as a matter of some urgency.

> [@Rajesh\_Kannan](#):
>
> Request: HEAD [http://myhost.mynw.com](http://myhost.mynw.com) /inp44olio-prdapp1-es/olio-elastic-prod/index-0

This is not AWS S3, it's some other allegedly-S3-compatible thing that turns out not to be properly compatible because ...

> [@Rajesh\_Kannan](#):
>
> "Caused by: java.net.SocketException: Connection reset",

... this is not how AWS S3 responds to this request. For further information, please see [these docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/repository-s3.html#repository-s3-compatible-services), particularly:

> You will need to work with the supplier of your storage system to address any incompatibilities you encounter. Please do not report Elasticsearch issues involving storage systems which claim to be S3-compatible unless you can demonstrate that the same issue exists when using a genuine AWS S3 repository.

---

<div class="post-metadata">

**Author:** ![Rajesh\_Kannan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rajesh_kannan/32/47154_2.png) [@Rajesh\_Kannan](https://discuss.elastic.co/u/Rajesh_Kannan)\
**Post date:** [December 27, 2024, 9:08am UTC](https://discuss.elastic.co/t/incremental-snapshots-are-failing-in-trying-to-write-new-repository-data-over-unfinished-write/372408/3 "2024-12-27T09:08:47Z")

</div>

> [@DavidTurner](#):
>
> llegedly-S3-compatible thing that turns out not to be

Hi David,

We were using selfhosted data storage named datacore. Also I checked the compatibility for this data storage and no issues found. Is there any fix for the current configuration?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [December 27, 2024, 9:15am UTC](https://discuss.elastic.co/t/incremental-snapshots-are-failing-in-trying-to-write-new-repository-data-over-unfinished-write/372408/4 "2024-12-27T09:15:01Z")

</div>

> [@Rajesh\_Kannan](#):
>
> Also I checked the compatibility for this data storage and no issues found.

This isn't the case, the problem you are reporting is a definite incompatibility with S3. Sorry, there's nothing we can do to help here, you need to ask your storage supplier to fix this.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [December 27, 2024, 9:51am UTC](https://discuss.elastic.co/t/incremental-snapshots-are-failing-in-trying-to-write-new-repository-data-over-unfinished-write/372408/5 "2024-12-27T09:51:02Z")

</div>

> [@DavidTurner](#):
>
> the problem you are reporting is a definite incompatibility with S3

Specifically, it does not properly support the [`HeadObject` API](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html). I've never seen anyone fail to get this right before, and we don't check for it in the [Repository Analyzer](https://www.elastic.co/guide/en/elasticsearch/reference/current/repo-analysis-api.html) so it may well pass those tests. I opened a feature request to add this extra check:

> <https://github.com/elastic/elasticsearch/issues/119303>
>
> \[This user\](https://discuss.elastic.co/t/incremental-snapshots-are-failing-in-tr…ying-to-write-new-repository-data-over-unfinished-write/372408) has a repository implementation that apparently does not correctly implement the \`HeadObject\` S3 API. Today's repository analyzer would not detect this incompatibility. We should fix that, for instance by having \`GetBlobChecksumAction\` sometimes (NB not always) call \`HeadObject\` before reading the blob contents.

---

<div class="post-metadata">

**Author:** ![Rajesh\_Kannan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rajesh_kannan/32/47154_2.png) [@Rajesh\_Kannan](https://discuss.elastic.co/u/Rajesh_Kannan)\
**Post date:** [January 3, 2025, 11:22am UTC](https://discuss.elastic.co/t/incremental-snapshots-are-failing-in-trying-to-write-new-repository-data-over-unfinished-write/372408/6 "2025-01-03T11:22:45Z")

</div>

Hi @DavidTurner ,

The incremental snapshots are failing, while the first snapshot is successful without any errors or issues. If there were an issue with the storage, the first snapshot should have failed as well. Kindly suggest if there are any configurations needs to be done to achieve incremental daily snapshots

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 3, 2025, 11:27am UTC](https://discuss.elastic.co/t/incremental-snapshots-are-failing-in-trying-to-write-new-repository-data-over-unfinished-write/372408/7 "2025-01-03T11:27:49Z")

</div>

When you create the first snapshot there is no existing data that need to be checked, so no need for the snapshot API to use the HeadObject API. Once you take subsequent snapshots the API need to check what is in the repo and I suspect this is where it fails due to the HeadObject API incompatibility. What you are describing therefore sound consistent with what David pointed out and that there is an issue with your storage API.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [January 3, 2025, 12:11pm UTC](https://discuss.elastic.co/t/incremental-snapshots-are-failing-in-trying-to-write-new-repository-data-over-unfinished-write/372408/8 "2025-01-03T12:11:20Z")

</div>

> [@Rajesh\_Kannan](#):
>
> If there were an issue with the storage, the first snapshot should have failed as well.

Yeah this simply isn't the case. The error is unambiguously a problem with your repository storage.

Please re-read the docs I linked [above](https://discuss.elastic.co/t/incremental-snapshots-are-failing-in-trying-to-write-new-repository-data-over-unfinished-write/372408/2), particularly

> Please do not report Elasticsearch issues involving storage systems which claim to be S3-compatible **unless you can demonstrate that the same issue exists when using a genuine AWS S3 repository**.

You need to try this with a genuine AWS S3 repository in order to see that it doesn't do the same thing, which proves your Datacore storage is not S3 compatible.