# IPv6 endpoint for S3 repository

**URL:** <https://discuss.elastic.co/t/ipv6-endpoint-for-s3-repository/259164>\
**Category:** Elasticsearch\
**Created:** [December 19, 2020, 3:42pm UTC](https://discuss.elastic.co/t/ipv6-endpoint-for-s3-repository/259164 "2020-12-19T15:42:16Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![NominaSumpta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nominasumpta/32/52412_2.png) [@NominaSumpta](https://discuss.elastic.co/u/NominaSumpta)\
**Post date:** [December 19, 2020, 3:42pm UTC](https://discuss.elastic.co/t/ipv6-endpoint-for-s3-repository/259164/1 "2020-12-19T15:42:16Z")

</div>

How do I use an IPv6 endpoint for an S3 repository?

Using a hostname that resolves to an IPv6 address and an IPv4 address results in:

```
{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[elasticsearch] path is not accessible on master node"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[elasticsearch] path is not accessible on master node",
    "caused_by": {
      "type": "i_o_exception",
      "reason": "Unable to upload object [tests-h__VhoSGSSaVgzOkkKWHqw/master.dat] using a single upload",
      "caused_by": {
        "type": "sdk_client_exception",
        "reason": "sdk_client_exception: Failed to connect to service endpoint: ",
        "caused_by": {
          "type": "i_o_exception",
          "reason": "Network is unreachable"
        }
      }
    }
  },
  "status": 500
}

```

So Elasticsearch is trying to connect with IPv4 (which is not the system preference, and which I don't have).

Using an IPv6 address results in:

```
{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[elasticsearch] path is not accessible on master node"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[elasticsearch] path is not accessible on master node",
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "Endpoint does not contain a valid host name: https://2a03:7900:1:3:31:3:104:123"
    }
  },
  "status": 500
}

```

Using brackets results in "u\_r\_i\_syntax\_exception: Illegal character in scheme name at index 0: [2a03:7900:1:3:31:3:104:123]".

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [December 19, 2020, 10:21pm UTC](https://discuss.elastic.co/t/ipv6-endpoint-for-s3-repository/259164/2 "2020-12-19T22:21:14Z")

</div>

The full stack trace is this:

```auto
[2020-12-19T21:58:02,857][WARN][r.suppressed] [node-0] path: /_snapshot/test, params: {repository=test}
org.elasticsearch.repositories.RepositoryVerificationException: [test] path is not accessible on master node
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1276) ~[elasticsearch-7.10.0.jar:7.10.0]
        at org.elasticsearch.repositories.RepositoriesService$3.doRun(RepositoriesService.java:268) ~[elasticsearch-7.10.0.jar:7.10.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) [elasticsearch-7.10.0.jar:7.10.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.0.jar:7.10.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: java.lang.IllegalArgumentException: Illegal character in scheme name at index 0: [2a03:7900:1:3:31:3:104:123]
        at java.net.URI.create(URI.java:906) ~[?:?]
        at com.amazonaws.regions.EndpointToRegion.guessRegionOrRegionNameForEndpoint(EndpointToRegion.java:71) ~[?:?]
        at com.amazonaws.regions.EndpointToRegion.guessRegionNameForEndpoint(EndpointToRegion.java:41) ~[?:?]
        at com.amazonaws.regions.EndpointToRegion.guessRegionNameForEndpointWithDefault(EndpointToRegion.java:48) ~[?:?]
        at com.amazonaws.AmazonWebServiceClient.computeSignerByURI(AmazonWebServiceClient.java:429) ~[?:?]
        at com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:318) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.setEndpoint(AmazonS3Client.java:725) ~[?:?]
        at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:453) ~[?:?]
        at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424) ~[?:?]
        at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
        at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:42) ~[?:?]
        at org.elasticsearch.repositories.s3.S3Service.buildClient(S3Service.java:164) ~[?:?]
        at org.elasticsearch.repositories.s3.S3Service.client(S3Service.java:96) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobStore.clientReference(S3BlobStore.java:125) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:355) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:131) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
        at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:48) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:129) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:146) ~[?:?]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1271) ~[elasticsearch-7.10.0.jar:7.10.0]
        ... 6 more
Caused by: java.net.URISyntaxException: Illegal character in scheme name at index 0: [2a03:7900:1:3:31:3:104:123]
        at java.net.URI$Parser.fail(URI.java:2938) ~[?:?]
        at java.net.URI$Parser.checkChars(URI.java:3109) ~[?:?]
        at java.net.URI$Parser.checkChar(URI.java:3119) ~[?:?]
        at java.net.URI$Parser.parse(URI.java:3134) ~[?:?]
        at java.net.URI.<init>(URI.java:623) ~[?:?]
        at java.net.URI.create(URI.java:904) ~[?:?]
        at com.amazonaws.regions.EndpointToRegion.guessRegionOrRegionNameForEndpoint(EndpointToRegion.java:71) ~[?:?]
        at com.amazonaws.regions.EndpointToRegion.guessRegionNameForEndpoint(EndpointToRegion.java:41) ~[?:?]
        at com.amazonaws.regions.EndpointToRegion.guessRegionNameForEndpointWithDefault(EndpointToRegion.java:48) ~[?:?]
        at com.amazonaws.AmazonWebServiceClient.computeSignerByURI(AmazonWebServiceClient.java:429) ~[?:?]
        at com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:318) ~[?:?]
        at com.amazonaws.services.s3.AmazonS3Client.setEndpoint(AmazonS3Client.java:725) ~[?:?]
        at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:453) ~[?:?]
        at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424) ~[?:?]
        at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:312) ~[?:?]
        at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:42) ~[?:?]
        at org.elasticsearch.repositories.s3.S3Service.buildClient(S3Service.java:164) ~[?:?]
        at org.elasticsearch.repositories.s3.S3Service.client(S3Service.java:96) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobStore.clientReference(S3BlobStore.java:125) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:355) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:131) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
        at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:48) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:129) ~[?:?]
        at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:146) ~[?:?]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1271) ~[elasticsearch-7.10.0.jar:7.10.0]
        ... 6 more

```

This looks like a bug and my guess is that it's in the AWS SDK rather than in Elasticsearch since there's some funky stuff going on in those `guessRegion...` methods, but it's hard to say for sure. Elasticsearch is definitely passing a valid URI for the endpoint.

It also sounds like something of a setup bug at your end to have a hostname that resolves to addresses at which the network in question is unreachable. Can you set up a hostname that resolves just to the IPv6 address? IMO that's a workaround that's worth trying.

---

<div class="post-metadata">

**Author:** ![NominaSumpta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nominasumpta/32/52412_2.png) [@NominaSumpta](https://discuss.elastic.co/u/NominaSumpta)\
**Post date:** [December 20, 2020, 8:23am UTC](https://discuss.elastic.co/t/ipv6-endpoint-for-s3-repository/259164/3 "2020-12-20T08:23:03Z")

</div>

> It also sounds like something of a setup bug at your end to have a hostname that resolves to addresses at which the network in question is unreachable.

The IPv4 address to which the hostname resolves is reachable (of course). But my clients - the Elasticsearch cluster nodes - are IPv6 only.

> Can you set up a hostname that resolves just to the IPv6 address? IMO that's a workaround that's worth trying.

Just tried by adding only the IPv6 address to `/etc/hosts`. That also results in "Network is unreachable".

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [December 20, 2020, 10:10am UTC](https://discuss.elastic.co/t/ipv6-endpoint-for-s3-repository/259164/4 "2020-12-20T10:10:35Z")

</div>

> [@NominaSumpta](#):
>
> The IPv4 address to which the hostname resolves is reachable (of course). But my clients - the Elasticsearch cluster nodes - are IPv6 only.

Yes, on reflection it should be skipping unreachable addresses, not failing outright.

> [@NominaSumpta](#):
>
> Just tried by adding only the IPv6 address to `/etc/hosts` . That also results in "Network is unreachable".

Hmm. To me that means either (a) it's using the IPv6 address you want it to, but finding out that this address is also on an unreachable network, or (b) it's ignoring `/etc/hosts` and using the IPv4 address anyway.

Perhaps a silly question, but have you confirmed that the endpoint is genuinely reachable from these hosts, using something else (e.g. `curl`)? If so, I think I'd try something like Wireshark to see what's really happening on the wire.

---

<div class="post-metadata">

**Author:** ![NominaSumpta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nominasumpta/32/52412_2.png) [@NominaSumpta](https://discuss.elastic.co/u/NominaSumpta)\
**Post date:** [December 20, 2020, 10:12am UTC](https://discuss.elastic.co/t/ipv6-endpoint-for-s3-repository/259164/5 "2020-12-20T10:12:41Z")

</div>

> Hmm. To me that means either (a) it's using the IPv6 address you want it to, but finding out that this address is also on an unreachable network, or (b) it's ignoring /etc/hosts and using the IPv4 address anyway.

I think you mean it's using the IPv4 address, right? The IPv6 address is reachable.

> Perhaps a silly question, but have you confirmed that the endpoint is genuinely reachable from these hosts, using something else (e.g. `curl` )? If so, I think I'd try something like Wireshark to see what's really happening on the wire.

I see a DNS lookup PUTting to `curl -X PUT https://$(hostname):9200/_snapshot/elasticsearch`, so `/etc/hosts` is being skipped.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [December 20, 2020, 10:17am UTC](https://discuss.elastic.co/t/ipv6-endpoint-for-s3-repository/259164/6 "2020-12-20T10:17:53Z")

</div>

> [@NominaSumpta](#):
>
> I think you mean it's using the IPv4 address, right? The IPv6 address is reachable.

No, I meant the IPv6 address, but ...

> [@NominaSumpta](#):
>
> I see a DNS lookup PUTting to `curl -X PUT https://$(hostname):9200/_snapshot/elasticsearch` , so `/etc/hosts` is being skipped.

... ok that'd be the problem then. Can you work out why it's ignoring `/etc/hosts`? Or set up a DNS entry that only has the `AAAA` record?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 17, 2021, 10:18am UTC](https://discuss.elastic.co/t/ipv6-endpoint-for-s3-repository/259164/7 "2021-01-17T10:18:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.