ECK + Istio mTLS

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
#1

I saw on the release notes a mention of using ECK with Istio with disabling HTTP level TLS - I've been trying to use mTLS with Istio and ES and running into problems both with the transport layer (which I think I've resolved by using network.bind_host = 127.0.0.1) and the ECK managed Kibana which seems to fail to communicate with the ES cluster when mTLS is enabled.

Is this supported, and if so is there any guidance on making this work?

Thanks!

#2

I don't believe we have an official guide for Istio or any other service meshes as there are many different ways in which they can be configured. It wouldn't be easy to figure out the exact cause of the problem without having access to the environment, but If you provide some details of your Istio setup and the Elasticsearch manifests, I can try to see if there's anything obvious that's missing from the configuration.

#3 
apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  annotations:
    common.k8s.elastic.co/controller-version: 1.0.0-beta1
  name: logging
spec:
  version: 7.4.0
  http:
    tls:
      selfSignedCertificate:
        disabled: true
  nodeSets:
  - name: master-node
    config:
      node.master: true
      node.data: false
      node.ingest: false
      node.ml: false
      node.store.allow_mmap: false
      index.store.type: niofs
      network.bind_host: 127.0.0.1
      xpack.monitoring.collection.enabled: true
      xpack.security.authc.realms:
        native:
          native1: 
            order: 1
    count: 3
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: openebs-hostpath
    podTemplate:
      metadata:
        annotations:
          "co.elastic.logs/module": elasticsearch
      spec:
        containers:
        - name: elasticsearch
          env:
          - name: ES_JAVA_OPTS
            value: -Xms4g -Xmx4g
          resources:
            requests:
              memory: 8Gi
              cpu: 0.5
            limits:
              memory: 8Gi
              cpu: 2
  - name: data-node
    config:
      node.data: true
      node.master: false
      node.store.allow_mmap: false
      index.store.type: niofs
      network.bind_host: 127.0.0.1
      xpack.monitoring.collection.enabled: true
      xpack.security.authc.realms:
        native:
          native1: 
            order: 1
    count: 5
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 400Gi
        storageClassName: openebs-hostpath
    podTemplate:
      metadata:
        annotations:
          "co.elastic.logs/module": elasticsearch
      spec:
        containers:
        - name: elasticsearch
          env:
          - name: ES_JAVA_OPTS
            value: -Xms8g -Xmx8g
          resources:
            requests:
              memory: 16Gi
              cpu: 0.5
            limits:
              memory: 16Gi
              cpu: 2
  secureSettings:
  - secretName: es-gcs-credentials

and

apiVersion: kibana.k8s.elastic.co/v1beta1
kind: Kibana
metadata:
  name: logging
spec:
  version: 7.4.0
  count: 1
  podTemplate:
    metadata:
      annotations:
        "co.elastic.logs/module": kibana
    spec:
      containers:
      - name: kibana
        resources:
          requests:
            memory: 1Gi
            cpu: 0.5
          limits:
            memory: 2Gi
            cpu: 2
  elasticsearchRef:
    name: logging
  config:
    kibana.defaultAppId: discover
  http:
    tls:
      selfSignedCertificate:
        disabled: true
#4

Thank you for reporting this. I did some tests and I think there are a couple of issues that prevent ECK-managed clusters from correctly working under Istio. I have created issues in the project repo to investigate them further.

#5

Awesome - I'll keep an eye on those issues then.

Thanks Charith

#6

You should be able to work around the issue with the transport port by defining an additional service for the transport layer so that Istio can discover the transport ports:

apiVersion: v1
kind: Service
metadata:
  labels:
    common.k8s.elastic.co/type: elasticsearch
    elasticsearch.k8s.elastic.co/cluster-name: $CLUSTER_NAME
  name: $CLUSTER_NAME-es-transport
  namespace: default
spec:
  clusterIP: none
  ports:
  - name: tcp-transport
    port: 9300
    protocol: TCP
    targetPort: 9300
  selector:
    common.k8s.elastic.co/type: elasticsearch
    elasticsearch.k8s.elastic.co/cluster-name: $CLUSTER_NAME
#7

Thanks Peter, I'll give that a test - also see the comment about Kibana log in loop when TLS is disabled. That sounds like a strong possibility as I've been testing quite a few different cluster deployments, I'll clear my browser and try that again.

Thanks!

#8

For anyone else interested, an example of using ECK 1.0.0-beta1 to deploy Elasticsearch and Kibana under Istio 1.3.3 can be found in https://github.com/elastic/cloud-on-k8s/issues/2064#issuecomment-547903913. Please note that this is general guidance only and not an official guide to ECK with Istio.