Incremental snapshots are failing in "Trying to write new repository data over unfinished write"

Elastic Stack Elasticsearch
1

The incremental snapshot is failing with the below error. There is only one successful backup created and the daily crons are failing with the below errors.

{"type": "server", "timestamp": "2024-12-26T06:46:46,097Z", "level": "INFO", "component": "o.e.r.b.BlobStoreRepository", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Trying to write new repository data over unfinished write, repo [backup-s3-repository] is at safe generation [0] and pending generation [2]", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,336Z", "level": "DEBUG", "component": "c.a.request", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Sending Request: HEAD http://myhost.mynw.com /inp44olio-prdapp1-es/olio-elastic-prod/index-0 Headers: (amz-sdk-invocation-id: f2ae0226-98f8-def1-64a9-9b84a192ac36, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium, ) ", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,336Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:0/0/500\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064646Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,336Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 String to Sign: '"AWS4-HMAC-SHA256\n20241226T064646Z\n20241226/us-east-1/s3/aws4_request\ne0e301b0381eb953db42bf35894c3ce4dcd3b6d39a4ef1d18f1b169b4b9009c5"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,337Z", "level": "DEBUG", "component": "c.a.request", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retrying Request: HEAD http://myhost.mynw.com /inp44olio-prdapp1-es/olio-elastic-prod/index-0 Headers: (amz-sdk-invocation-id: f2ae0226-98f8-def1-64a9-9b84a192ac36, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium, ) ", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,337Z", "level": "DEBUG", "component": "c.a.h.AmazonHttpClient", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retriable error detected, will retry in 14ms, attempt number: 0", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,370Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:1/14/495\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064646Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,370Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 String to Sign: '"AWS4-HMAC-SHA256\n20241226T064646Z\n20241226/us-east-1/s3/aws4_request\n634de39f28bb52495cf02d05954c4285b1bdb0639777dce1cda01efcdfdd495b"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,371Z", "level": "DEBUG", "component": "c.a.request", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retrying Request: HEAD http://myhost.mynw.com /inp44olio-prdapp1-es/olio-elastic-prod/index-0 Headers: (amz-sdk-invocation-id: f2ae0226-98f8-def1-64a9-9b84a192ac36, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium, ) ", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,372Z", "level": "DEBUG", "component": "c.a.h.AmazonHttpClient", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retriable error detected, will retry in 15ms, attempt number: 1", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,387Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:2/15/490\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064646Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,387Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 String to Sign: '"AWS4-HMAC-SHA256\n20241226T064646Z\n20241226/us-east-1/s3/aws4_request\nc7894cb9600d567a56fa0164e1aec018e1b1918a7669207de47bad05d6ef4b9c"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,388Z", "level": "DEBUG", "component": "c.a.request", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retrying Request: HEAD http://myhost.mynw.com /inp44olio-prdapp1-es/olio-elastic-prod/index-0 Headers: (amz-sdk-invocation-id: f2ae0226-98f8-def1-64a9-9b84a192ac36, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium, ) ", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,388Z", "level": "DEBUG", "component": "c.a.h.AmazonHttpClient", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retriable error detected, will retry in 342ms, attempt number: 2", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,730Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:3/342/485\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064646Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,731Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 String to Sign: '"AWS4-HMAC-SHA256\n20241226T064646Z\n20241226/us-east-1/s3/aws4_request\nfe860cd2099cb03de0c9eccaa1a4d98f93ee11fc347101ba1f601c3c64d06a16"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,732Z", "level": "DEBUG", "component": "c.a.request", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retrying Request: HEAD http://myhost.mynw.com /inp44olio-prdapp1-es/olio-elastic-prod/index-0 Headers: (amz-sdk-invocation-id: f2ae0226-98f8-def1-64a9-9b84a192ac36, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium, ) ", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,732Z", "level": "DEBUG", "component": "c.a.h.AmazonHttpClient", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retriable error detected, will retry in 113ms, attempt number: 3", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,845Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:4/113/480\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064646Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,846Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 String to Sign: '"AWS4-HMAC-SHA256\n20241226T064646Z\n20241226/us-east-1/s3/aws4_request\n63daafe677d4a23c7212b71ed12ef82cbf59ab0d98cb584511194dcd58a390b8"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,856Z", "level": "DEBUG", "component": "c.a.request", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retrying Request: HEAD http://myhost.mynw.com /inp44olio-prdapp1-es/olio-elastic-prod/index-0 Headers: (amz-sdk-invocation-id: f2ae0226-98f8-def1-64a9-9b84a192ac36, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium, ) ", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:46,856Z", "level": "DEBUG", "component": "c.a.h.AmazonHttpClient", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retriable error detected, will retry in 659ms, attempt number: 4", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:47,515Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:5/659/475\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064647Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:47,517Z", "level": "DEBUG", "component": "c.a.h.AmazonHttpClient", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retriable error detected, will retry in 1582ms, attempt number: 5", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:49,099Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:6/1582/470\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064649Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:49,099Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 String to Sign: '"AWS4-HMAC-SHA256\n20241226T064649Z\n20241226/us-east-1/s3/aws4_request\n3967b1a30aa706f881e979c7b3f0a7b7d35a450be452a76dfebd668dbd8cb948"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:49,102Z", "level": "DEBUG", "component": "c.a.request", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retrying Request: HEAD http://myhost.mynw.com /inp44olio-prdapp1-es/olio-elastic-prod/index-0 Headers: (amz-sdk-invocation-id: f2ae0226-98f8-def1-64a9-9b84a192ac36, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium, ) ", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:49,102Z", "level": "DEBUG", "component": "c.a.h.AmazonHttpClient", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retriable error detected, will retry in 1262ms, attempt number: 6", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:50,364Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:7/1262/465\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064650Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:50,365Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 String to Sign: '"AWS4-HMAC-SHA256\n20241226T064650Z\n20241226/us-east-1/s3/aws4_request\nffddd6496d90f82da3e84ea8cc841f58da1b3bca8a54db63f23e7c65aaf752e8"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:50,366Z", "level": "DEBUG", "component": "c.a.request", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retrying Request: HEAD http://myhost.mynw.com /inp44olio-prdapp1-es/olio-elastic-prod/index-0 Headers: (amz-sdk-invocation-id: f2ae0226-98f8-def1-64a9-9b84a192ac36, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium, ) ", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:50,366Z", "level": "DEBUG", "component": "c.a.h.AmazonHttpClient", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retriable error detected, will retry in 5619ms, attempt number: 7", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:55,986Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:8/5619/460\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064655Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:55,986Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 String to Sign: '"AWS4-HMAC-SHA256\n20241226T064655Z\n20241226/us-east-1/s3/aws4_request\n87f579bee63261074c6bbf38f43bc516d63db09df406480ac6ed02efd142d074"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:55,988Z", "level": "DEBUG", "component": "c.a.request", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retrying Request: HEAD http://myhost.mynw.com /inp44olio-prdapp1-es/olio-elastic-prod/index-0 Headers: (amz-sdk-invocation-id: f2ae0226-98f8-def1-64a9-9b84a192ac36, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium, ) ", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:46:55,988Z", "level": "DEBUG", "component": "c.a.h.AmazonHttpClient", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retriable error detected, will retry in 19823ms, attempt number: 8", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:47:05,490Z", "level": "INFO", "component": "o.e.x.s.a.AuthorizationService", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Took [70ms] to resolve [4063] indices for action [indices:data/read/search] and user [olio-prod]", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:47:15,811Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:9/19823/455\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064715Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:47:15,811Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 String to Sign: '"AWS4-HMAC-SHA256\n20241226T064715Z\n20241226/us-east-1/s3/aws4_request\n408e57144f9f163f607670f1ec7e5b834287c62ff443ba68fd7d1b5cc06c2482"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:47:15,813Z", "level": "DEBUG", "component": "c.a.request", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retrying Request: HEAD http://myhost.mynw.com /inp44olio-prdapp1-es/olio-elastic-prod/index-0 Headers: (amz-sdk-invocation-id: f2ae0226-98f8-def1-64a9-9b84a192ac36, Content-Type: application/octet-stream, User-Agent: aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium, ) ", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:47:15,813Z", "level": "DEBUG", "component": "c.a.h.AmazonHttpClient", "cluster.name": "es", "node.name": "es-es-master-0", "message": "Retriable error detected, will retry in 99ms, attempt number: 9", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:47:15,912Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 Canonical Request: '"HEAD\n/inp44olio-prdapp1-es/olio-elastic-prod/index-0\n\namz-sdk-invocation-id:f2ae0226-98f8-def1-64a9-9b84a192ac36\namz-sdk-retry:10/99/450\ncontent-type:application/octet-stream\nhost:myhost.mynw.com\nuser-agent:aws-sdk-java/1.11.749 Linux/5.15.0-125-generic OpenJDK_64-Bit_Server_VM/17.0.1+12 java/17.0.1 vendor/Eclipse_Adoptium\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241226T064715Z\n\namz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:47:15,913Z", "level": "DEBUG", "component": "c.a.a.AWS4Signer", "cluster.name": "es", "node.name": "es-es-master-0", "message": "AWS4 String to Sign: '"AWS4-HMAC-SHA256\n20241226T064715Z\n20241226/us-east-1/s3/aws4_request\n405d2ce8360ad3d14231bcfbe04c0c20f355b46899a81c28a55383ea3d14c63f"", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" }
{"type": "server", "timestamp": "2024-12-26T06:47:15,915Z", "level": "WARN", "component": "o.e.s.SnapshotsService", "cluster.name": "es", "node.name": "es-es-master-0", "message": "[backup-s3-repository:snapshot-202412260622/0lwk886URJes3jGDS-hrMQ] failed to finalize snapshot", "cluster.uuid": "leysDoepT1GnwqcNlLTBhA", "node.id": "v3Lvq4kDRv6vkgSaxsg6PQ" ,
"stacktrace": ["org.elasticsearch.snapshots.SnapshotException: [backup-s3-repository:snapshot-202412260622/0lwk886URJes3jGDS-hrMQ] failed to update snapshot in repository",
"at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$42(BlobStoreRepository.java:1377) [elasticsearch-7.16.2.jar:7.16.2]",
"at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:144) [elasticsearch-7.16.2.jar:7.16.2]",
"at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:144) [elasticsearch-7.16.2.jar:7.16.2]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:138) [elasticsearch-7.16.2.jar:7.16.2]",
"at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) [elasticsearch-7.16.2.jar:7.16.2]",
"at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:100) [elasticsearch-7.16.2.jar:7.16.2]",
"at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:131) [elasticsearch-7.16.2.jar:7.16.2]",
"at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:139) [elasticsearch-7.16.2.jar:7.16.2]",
"at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:52) [elasticsearch-7.16.2.jar:7.16.2]",
"at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:29) [elasticsearch-7.16.2.jar:7.16.2]",
"at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$63(BlobStoreRepository.java:2300) [elasticsearch-7.16.2.jar:7.16.2]",
"at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) [elasticsearch-7.16.2.jar:7.16.2]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.16.2.jar:7.16.2]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.16.2.jar:7.16.2]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]",
"at java.lang.Thread.run(Thread.java:833) [?:?]",
"Caused by: org.elasticsearch.common.blobstore.BlobStoreException: Failed to check if blob [index-0] exists",
"at org.elasticsearch.repositories.s3.S3BlobContainer.blobExists(S3BlobContainer.java:90) ~[?:?]",
"at org.elasticsearch.repositories.blobstore.BlobStoreRepository.ensureSafeGenerationExists(BlobStoreRepository.java:2462) ~[elasticsearch-7.16.2.jar:7.16.2]",
"at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$66(BlobStoreRepository.java:2316) ~[elasticsearch-7.16.2.jar:7.16.2]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.16.2.jar:7.16.2]",
"... 13 more",
"Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connection reset",
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1201) ~[?:?]",
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1147) ~[?:?]",
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796) ~[?:?]",
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) ~[?:?]",
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) ~[?:?]",
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) ~[?:?]",
"at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) ~[?:?]",
"at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) ~[?:?]",
"at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) ~[?:?]",
"at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.16.2.jar:7.16.2]",
"... 13 more",
"Caused by: java.net.SocketException: Connection reset",
"at sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:323) ~[?:?]",
"at sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350) ~[?:?]",
"at sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803) ~[?:?]",
"at java.net.Socket$SocketInputStream.read(Socket.java:966) ~[?:?]",
"at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[?:?]",
"at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[?:?]",
"at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) ~[?:?]",
"at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[?:?]",
"at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[?:?]",
"at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[?:?]",
"at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[?:?]",
"at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) ~[?:?]",
"at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[?:?]",
"at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:82) ~[?:?]",
"at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[?:?]",
"at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[?:?]",
"at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[?:?]",
"at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[?:?]",
"at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[?:?]",
"at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[?:?]",
"at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[?:?]",
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323) ~[?:?]",
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139) ~[?:?]",
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796) ~[?:?]",
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) ~[?:?]",
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) ~[?:?]",
"at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) ~[?:?]",
"at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) ~[?:?]",
"at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) ~[?:?]",
"at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) ~[?:?]",
"at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054) ~[?:?]",
"at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000) ~[?:?]",
"at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1335) ~[?:?]",
"at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1309) ~[?:?]",
"at com.amazonaws.services.s3.AmazonS3Client.doesObjectExist(AmazonS3Client.java:1390) ~[?:?]",
"at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$blobExists$0(S3BlobContainer.java:88) ~[?:?]",
"at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]",
"at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) ~[?:?]",
"at org.elasticsearch.repositories.s3.S3BlobContainer.blobExists(S3BlobContainer.java:88) ~[?:?]",
"at org.elasticsearch.repositories.blobstore.BlobStoreRepository.ensureSafeGenerationExists(BlobStoreRepository.java:2462) ~[elasticsearch-7.16.2.jar:7.16.2]",
"at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$66(BlobStoreRepository.java:2316) ~[elasticsearch-7.16.2.jar:7.16.2]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.16.2.jar:7.16.2]",

2

This version is very old and no longer supported. You need to upgrade (ideally to an 8.x version) as a matter of some urgency.

This is not AWS S3, it's some other allegedly-S3-compatible thing that turns out not to be properly compatible because ...

... this is not how AWS S3 responds to this request. For further information, please see these docs, particularly:

You will need to work with the supplier of your storage system to address any incompatibilities you encounter. Please do not report Elasticsearch issues involving storage systems which claim to be S3-compatible unless you can demonstrate that the same issue exists when using a genuine AWS S3 repository.

1 Like
3

Hi David,

We were using selfhosted data storage named datacore. Also I checked the compatibility for this data storage and no issues found. Is there any fix for the current configuration?

4

This isn't the case, the problem you are reporting is a definite incompatibility with S3. Sorry, there's nothing we can do to help here, you need to ask your storage supplier to fix this.

5

Specifically, it does not properly support the HeadObject API. I've never seen anyone fail to get this right before, and we don't check for it in the Repository Analyzer so it may well pass those tests. I opened a feature request to add this extra check:

6

Hi @DavidTurner ,

The incremental snapshots are failing, while the first snapshot is successful without any errors or issues. If there were an issue with the storage, the first snapshot should have failed as well. Kindly suggest if there are any configurations needs to be done to achieve incremental daily snapshots

7

When you create the first snapshot there is no existing data that need to be checked, so no need for the snapshot API to use the HeadObject API. Once you take subsequent snapshots the API need to check what is in the repo and I suspect this is where it fails due to the HeadObject API incompatibility. What you are describing therefore sound consistent with what David pointed out and that there is an issue with your storage API.

1 Like
8

Yeah this simply isn't the case. The error is unambiguously a problem with your repository storage.

Please re-read the docs I linked above, particularly

Please do not report Elasticsearch issues involving storage systems which claim to be S3-compatible unless you can demonstrate that the same issue exists when using a genuine AWS S3 repository.

You need to try this with a genuine AWS S3 repository in order to see that it doesn't do the same thing, which proves your Datacore storage is not S3 compatible.

1 Like