IPv6 endpoint for S3 repository

Elastic Stack Elasticsearch
1

How do I use an IPv6 endpoint for an S3 repository?

Using a hostname that resolves to an IPv6 address and an IPv4 address results in:

{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[elasticsearch] path  is not accessible on master node"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[elasticsearch] path  is not accessible on master node",
    "caused_by": {
      "type": "i_o_exception",
      "reason": "Unable to upload object [tests-h__VhoSGSSaVgzOkkKWHqw/master.dat] using a single upload",
      "caused_by": {
        "type": "sdk_client_exception",
        "reason": "sdk_client_exception: Failed to connect to service endpoint: ",
        "caused_by": {
          "type": "i_o_exception",
          "reason": "Network is unreachable"
        }
      }
    }
  },
  "status": 500
}

So Elasticsearch is trying to connect with IPv4 (which is not the system preference, and which I don't have).

Using an IPv6 address results in:

{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[elasticsearch] path  is not accessible on master node"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[elasticsearch] path  is not accessible on master node",
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "Endpoint does not contain a valid host name: https://2a03:7900:1:3:31:3:104:123"
    }
  },
  "status": 500
}

Using brackets results in "u_r_i_syntax_exception: Illegal character in scheme name at index 0: [2a03:7900:1:3:31:3:104:123]".

2

The full stack trace is this:

[2020-12-19T21:58:02,857][WARN ][r.suppressed             ] [node-0] path: /_snapshot/test, params: {repository=test}
org.elasticsearch.repositories.RepositoryVerificationException: [test] path  is not accessible on master node
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1276) ~[elasticsearch-7.10.0.jar:7.10.0]
        at org.elasticsearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:268) ~[elasticsearch-7.10.0.jar:7.10.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) [elasticsearch-7.10.0.jar:7.10.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.0.jar:7.10.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: java.lang.IllegalArgumentException: Illegal character in scheme name at index 0: [2a03:7900:1:3:31:3:104:123]
        at java.net.URI.create(URI.java:906) ~[?:?]
        at com.amazonaws.regions.EndpointToRegion.guessRegionOrRegionNameForEndpoint(EndpointToRegion.java:71) ~[?:?]
        at com.amazonaws.regions.EndpointToRegion.guessRegionNameForEndpoint(EndpointToRegion.java:41) ~[?:?]
        at com.amazonaws.regions.EndpointToRegion.guessRegionNameForEndpointWithDefault(EndpointToRegion.java:48) ~[?:?]
        at com.amazonaws.AmazonWebServiceClient.computeSignerByURI(AmazonWebServiceClient.java:429) ~[?:?]
        at com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:318) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.setEndpoint(AmazonS3Client.java:725) ~[?:?]
        at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:453) ~[?:?]
        at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424) ~[?:?]
        at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
        at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:42) ~[?:?]
        at org.elasticsearch.repositories.s3.S3Service.buildClient(S3Service.java:164) ~[?:?]
        at org.elasticsearch.repositories.s3.S3Service.client(S3Service.java:96) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobStore.clientReference(S3BlobStore.java:125) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:355) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:131) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
        at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:48) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:129) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:146) ~[?:?]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1271) ~[elasticsearch-7.10.0.jar:7.10.0]
        ... 6 more
Caused by: java.net.URISyntaxException: Illegal character in scheme name at index 0: [2a03:7900:1:3:31:3:104:123]
        at java.net.URI$Parser.fail(URI.java:2938) ~[?:?]
        at java.net.URI$Parser.checkChars(URI.java:3109) ~[?:?]
        at java.net.URI$Parser.checkChar(URI.java:3119) ~[?:?]
        at java.net.URI$Parser.parse(URI.java:3134) ~[?:?]
        at java.net.URI.<init>(URI.java:623) ~[?:?]
        at java.net.URI.create(URI.java:904) ~[?:?]
        at com.amazonaws.regions.EndpointToRegion.guessRegionOrRegionNameForEndpoint(EndpointToRegion.java:71) ~[?:?]
        at com.amazonaws.regions.EndpointToRegion.guessRegionNameForEndpoint(EndpointToRegion.java:41) ~[?:?]
        at com.amazonaws.regions.EndpointToRegion.guessRegionNameForEndpointWithDefault(EndpointToRegion.java:48) ~[?:?]
        at com.amazonaws.AmazonWebServiceClient.computeSignerByURI(AmazonWebServiceClient.java:429) ~[?:?]
        at com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:318) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.setEndpoint(AmazonS3Client.java:725) ~[?:?]
        at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:453) ~[?:?]
        at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424) ~[?:?]
        at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
        at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:42) ~[?:?]
        at org.elasticsearch.repositories.s3.S3Service.buildClient(S3Service.java:164) ~[?:?]
        at org.elasticsearch.repositories.s3.S3Service.client(S3Service.java:96) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobStore.clientReference(S3BlobStore.java:125) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:355) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:131) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
        at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:48) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:129) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:146) ~[?:?]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1271) ~[elasticsearch-7.10.0.jar:7.10.0]
        ... 6 more

This looks like a bug and my guess is that it's in the AWS SDK rather than in Elasticsearch since there's some funky stuff going on in those guessRegion... methods, but it's hard to say for sure. Elasticsearch is definitely passing a valid URI for the endpoint.

It also sounds like something of a setup bug at your end to have a hostname that resolves to addresses at which the network in question is unreachable. Can you set up a hostname that resolves just to the IPv6 address? IMO that's a workaround that's worth trying.

3

It also sounds like something of a setup bug at your end to have a hostname that resolves to addresses at which the network in question is unreachable.

The IPv4 address to which the hostname resolves is reachable (of course). But my clients - the Elasticsearch cluster nodes - are IPv6 only.

Can you set up a hostname that resolves just to the IPv6 address? IMO that's a workaround that's worth trying.

Just tried by adding only the IPv6 address to /etc/hosts. That also results in "Network is unreachable".

4

Yes, on reflection it should be skipping unreachable addresses, not failing outright.

Hmm. To me that means either (a) it's using the IPv6 address you want it to, but finding out that this address is also on an unreachable network, or (b) it's ignoring /etc/hosts and using the IPv4 address anyway.

Perhaps a silly question, but have you confirmed that the endpoint is genuinely reachable from these hosts, using something else (e.g. curl)? If so, I think I'd try something like Wireshark to see what's really happening on the wire.

5

Hmm. To me that means either (a) it's using the IPv6 address you want it to, but finding out that this address is also on an unreachable network, or (b) it's ignoring /etc/hosts and using the IPv4 address anyway.

I think you mean it's using the IPv4 address, right? The IPv6 address is reachable.

Perhaps a silly question, but have you confirmed that the endpoint is genuinely reachable from these hosts, using something else (e.g. curl )? If so, I think I'd try something like Wireshark to see what's really happening on the wire.

I see a DNS lookup PUTting to curl -X PUT https://$(hostname):9200/_snapshot/elasticsearch, so /etc/hosts is being skipped.

6

No, I meant the IPv6 address, but ...

... ok that'd be the problem then. Can you work out why it's ignoring /etc/hosts? Or set up a DNS entry that only has the AAAA record?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.