DNS not working on Cross Cluster Search?

[xamox]

Trying to setup cross cluster search with DNS and it appears to not be working, but working fine if I specify IP.

I'm using Elasticsearch 6.1.1

I've confirmed the DNS is propagated and is mapped to the correct IP address:

DNS propagated and working:
dig +short a elasticsearch-master.dev-k8s-uw1-gcp.REDACTED.com
10.138.0.58

Setup using IP:

sh-4.2# curl localhost:9200/_remote/info?pretty
{
  "dev_gcp" : {
    "seeds" : [
      "10.138.0.58:9300"
    ],
    "http_addresses" : [
      "10.36.21.4:9200",
      "10.36.24.23:9200",
      "10.36.25.15:9200"
    ],
    "connected" : true,
    "num_nodes_connected" : 3,
    "max_connections_per_cluster" : 3,
    "initial_connect_timeout" : "30s",
    "skip_unavailable" : true
  }
}

Working with test search:

sh-4.2# curl -m 20 localhost:9200/dev_gcp:logstash*/_search?pretty
{
  "took" : 88,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "_clusters" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "dev_gcp:logstash",
        "_type" : "federationtest",
        "_id" : "JBhVhGMBoxmXoJByyoQS",
        "_score" : 1.0,
        "_source" : {
          "field" : "value"
        }
      }
    ]
  }
}

If I remove the remote cluster

sh-4.2# curl localhost:9200/_cluster/settings?pretty
{
  "persistent" : { },
  "transient" : { }
}

Then add it back with DNS:

curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "search": {
      "remote": {
        "dev_gcp": {
          "seeds": "elasticsearch-master.dev-k8s-uw1-gcp.REDACTED.com:9300",
          "skip_unavailable": "true"
        }
      }
    }
  }
}
'

Confirm it's added:

sh-4.2# curl localhost:9200/_cluster/settings?pretty
{
  "persistent" : {
    "search" : {
      "remote" : {
        "dev_gcp" : {
          "skip_unavailable" : "true",
          "seeds" : "elasticsearch-master.dev-k8s-uw1-gcp.REDACTED.com:9300"
        }
      }
    }
  },
  "transient" : { }
}

But then look at the remote info:

sh-4.2# curl localhost:9200/_remote/info?pretty
{
  "dev_gcp" : {
    "seeds" : [ ],
    "http_addresses" : [ ],
    "connected" : false,
    "num_nodes_connected" : 0,
    "max_connections_per_cluster" : 3,
    "initial_connect_timeout" : "30s",
    "skip_unavailable" : true
  }
}

It get's nothing.

But I know that domain is routable from that node, confirmed with netcat:

sh-4.2# nc -vvvvvv elasticsearch-master.dev-k8s-uw1-gcp.REDACTED.com 9300
Ncat: Version 6.40 ( http://nmap.org/ncat )
libnsock nsi_new2(): nsi_new (IOD #1)
libnsock msevent_new(): msevent_new (IOD #1) (EID #8)
libnsock nsock_connect_tcp(): TCP connection requested to 10.138.0.58:9300 (IOD #1) EID 8
libnsock nsp_add_event(): NSE #8: Adding event
libnsock nsock_loop(): nsock_loop() started (no timeout). 1 events pending
libnsock nsock_trace_handler_callback(): Callback: CONNECT SUCCESS for EID 8 [10.138.0.58:9300]
Ncat: Connected to 10.138.0.58:9300.
libnsock nsi_new2(): nsi_new (IOD #2)
libnsock msevent_new(): msevent_new (IOD #1) (EID #18)
libnsock nsock_read(): Read request from IOD #1 [10.138.0.58:9300] (timeout: -1ms) EID 18
libnsock nsp_add_event(): NSE #18: Adding event
libnsock msevent_new(): msevent_new (IOD #2) (EID #26)
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 26
libnsock nsp_add_event(): NSE #26: Adding event
libnsock msevent_delete(): msevent_delete (IOD #1) (EID #8)

Any help would be great. Not sure why DNS wouldn't work.

[xamox]