Remote cluster - TCP connection is not happened with Istio ingress

[rmssath]

Hi Team,

We have followed the below step to configure remote cluster.

Remote clusters | Elastic Cloud on Kubernetes [1.5] | Elastic

In remote cluster, we have create istio ingress gateway/virtualservice with TCP to expose the service.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: es-gateway
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 9500
      name: hello1
      protocol: TCP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: es-vs
spec:
  hosts:
  - "*"
  gateways:
  - es-gateway
  tcp:
  - match:
    - port: 9500
    route:
    - destination:
        port:
          number: 9300
        host: quickstart-es-transport.default.svc.cluster.local

After configure remote cluster, we are getting request in remote cluster (istio logs) like below which is failed to connect.

Note: x.x.x.x is remote cluster

[2021-04-22T13:02:59.554Z] "- - -" 0 UH "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - x.x.x.x:9500 y.y.y.y:46395 - -
[2021-04-22T13:02:59.554Z] "- - -" 0 UH "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - x.x.x.x:9500 y.y.y.y:29463 - -
[2021-04-22T13:02:59.554Z] "- - -" 0 UH "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - x.x.x.x:9500 y.y.y.y:9801 - -
[2021-04-22T13:02:59.554Z] "- - -" 0 UH "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - x.x.x.x:9500 y.y.y.y:53429 - -
[2021-04-22T13:02:59.554Z] "- - -" 0 UH "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - x.x.x.x:9500 y.y.y.y:35396 - -

[Thibault_Richard (Thibault)]

Hello,

I wonder if the issue comes from the remote cluster setup or the Istio setup. Could you first try without Istio to better isolate the problem?

[rmssath]

Thanks Richard..

We are able to access the remote cluster with nodeport. Above issue is occur only on configuring with istio.

Is any changes or additional config required in above virtualservice?

[Thibault_Richard (Thibault)]

You have to exclude the transport port (port 9300) from being proxied. Currently ECK does not support switching off X-Pack security and TLS for the Elasticsearch transport port. If Istio is allowed to proxy the transport port, the traffic is encrypted twice and communication between Elasticsearch nodes is disrupted.

See the documentation: Istio | Elastic Cloud on Kubernetes [1.5] | Elastic.
Check the recipes directory in the ECK source repository for a complete example.