Cannot disable TLS and security in EKS

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hello,

I Installed ECK open source 1.0.1 in k8s 1.15.5
I tried to disable the security and TLS using:

  apiVersion: elasticsearch.k8s.elastic.co/v1
  kind: Elasticsearch
  metadata:
    name: elasticsearch
  spec:
    version: 7.6.0
    nodeSets:
    - name: default
      count: 1
      config:
        node.master: true
        node.data: true
        node.ingest: true
        node.store.allow_mmap: false

        xpack.security.enabled: false
        xpack.security.http.ssl.enabled: false
        xpack.security.transport.ssl.enabled: false

but it doesn't seem to work:

kubectl port-forward service/elasticsearch-es-http 9200
curl http://localhost:9200
==> not working

and

PASSWORD=$(kubectl get secret elasticsearch-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode)
curl -k -u "elastic:$PASSWORD" https://localhost:9200
==> works

If I check the configuration in the 'elasticsearch-es-default-es-config'
kubectl get secrets elasticsearch-es-default-es-config -o jsonpath="{.data['elasticsearch.yml']}" | base64 -d
I get:

cluster:
  name: elasticsearch
discovery:
  seed_providers: file
network:
  host: 0.0.0.0
  publish_host: ${POD_IP}
node:
  data: true
  ingest: true
  master: true
  name: ${POD_NAME}
  store:
    allow_mmap: false
path:
  data: /usr/share/elasticsearch/data
  logs: /usr/share/elasticsearch/logs
xpack:
  license:
    upload:
      types:
      - trial
      - enterprise
  security:
    authc:
      realms:
        file:
          file1:
            order: -100
        native:
          native1:
            order: -99
      reserved_realm:
        enabled: "false"
    enabled: "true"
    http:
      ssl:
        certificate: /usr/share/elasticsearch/config/http-certs/tls.crt
        certificate_authorities: /usr/share/elasticsearch/config/http-certs/ca.crt
        enabled: true
        key: /usr/share/elasticsearch/config/http-certs/tls.key
    transport:
      ssl:
        certificate: /usr/share/elasticsearch/config/node-transport-cert/transport.tls.crt
        certificate_authorities:
        - /usr/share/elasticsearch/config/transport-certs/ca.crt
        enabled: "true"
        key: /usr/share/elasticsearch/config/node-transport-cert/transport.tls.key
        verification_mode: certificate

which still contains the security and tls config enabled...

Any idea on how to disable security and tls config ?
Thanks
Yann

2

These settings are managed by ECK and you cannot currently disable security and you cannot disable TLS on the transport layer. Please see Settings managed by ECK | Elastic Cloud on Kubernetes [2.10] | Elastic for a list of settings managed by ECK that we don't support to be set by users.

If you want to disable security in order to allow access without authentication to your clusters, please consider enabling anonymous access instead: Enabling anonymous access | Elasticsearch Guide [7.6] | Elastic

You can however disable TLS on the HTTP layer if desired. Instructions are here: Access Elastic Stack services | Elastic Cloud on Kubernetes [2.10] | Elastic

3

Thanks Peter, that solved my issue.

for folks here is the complete solution:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  version: 7.6.0
  nodeSets:
  - name: default
    count: 1
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
      xpack.security.authc:
          anonymous:
            username: anonymous
            roles: superuser
            authz_exception: false
  http:
    tls:
      selfSignedCertificate:
        disabled: true

Yann

1 Like
4

Hi, I tried this but it doesn't seems to work for ES.

5

Thank you for the instructions. But it seems like the option for disabling TLS on http layer must be set when the ES cluster is initially deployed. I cannot disable that on an existing cluster with TLS enabled. The ECK would not make that switch even I added that section:

http:
    tls:
      certificate:
        selfSignedCertificate:
          disabled: true
1 Like
6

hi, I can confirm that applying this config and then removing the default pod does not seem to work, it keeps https. Any solutions to that?

More specifially, it says here:

how it should be disabled.

here it says xpack security is disabled by default for non-trial:

that does not seem to be the case for k8s.
What is the recommended way for k8s to disable https?
Seems like a more sane default too, since most folks probably expose https via ingress-nginx or an external Google Loadbalancer?

7

What is the recommended way for k8s to disable https?

You have pointed the correct documentation: TLS certificates | Elastic Cloud on Kubernetes [2.10] | Elastic.

here it says xpack security is disabled by default for non-trial

This is not the ECK documentation, so it does not apply to ECK.

Here is an example step by step on how to disable TLS on the HTTP layer of an existing Elasticsearch cluster managed by ECK.

On a fresh GKE v1.18.14 cluster, I deployed ECK v1.3.1 and a simple Elasticsearch cluster using this manifest:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: badaboum
spec:
  version: 7.10.2
  nodeSets:
  - name: master
    count: 1
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false

Started the port-forwarding:

kubectl port-forward service/badaboum-es-http 9200

Checked that HTTPS was working but not HTTP:

> curl -k https://localhost:9200
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer real
m=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer 
realm=\"security\"","ApiKey"]}},"status":401} 

> curl http://localhost:9200
curl: (52) Empty reply from server

Then, I updated my Elasticsearch cluster to disable TLS with this manifest:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: badaboum
spec:
  version: 7.10.2
  http:                  
    tls:
      selfSignedCertificate:
        disabled: true
  nodeSets:
  - name: master
    count: 1
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false

Restarted the port-forwarding.

Checked that HTTPS was no longer working and HTTP was working:

> curl -k https://localhost:9200
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

> curl http://localhost:9200
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"se
curity_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}
8

hi, thank you for your reply. Isn't that the exact same snippet for the Elasticsearch file?
Update : I had a snippet added which had a typo, certificate without capital letter.
After correcting that and deleting the pod it works as expected, thanks for yoru support!

9

Hello @Zijun_X ,

Now I see the error in your snippet.

You shared:

http:
    tls:
      certificate:
        selfSignedCertificate:
          disabled: true

While the 'certificate' field shouldn't be present:

http:
    tls:
      selfSignedCertificate:
        disabled: true
3 Likes
10

Same issue.
After add selfSignedCertificate into ES.
Search by http is correct.

# curl -u "elastic:$PASSWORD" -k "http://xxxxxx:30300"
{
  "name" : "es-monitor-es-master-1",
  "cluster_name" : "es-monitor",
  "cluster_uuid" : "jxYqF_E0QuSAzQsJII63xQ",
  "version" : {
    "number" : "7.14.1",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "66b55ebfa59c92c15db3f69a335d500018b3331e",
    "build_date" : "2021-08-26T09:01:05.390870785Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

But can not connect from kibana.
es_monitor.yaml

# cat es_monitor.yaml
apiVersion: v1
items:
- apiVersion: elasticsearch.k8s.elastic.co/v1
  kind: Elasticsearch
  metadata:
    annotations:
      common.k8s.elastic.co/controller-version: 1.7.1
      eck.k8s.elastic.co/es-client-timeout: 60s
    name: es-monitor
    namespace: monitor
  spec:
    auth: {}
    http:
      service:
        metadata: {}
        spec:
          ports:
          - name: https
            nodePort: 30300
            port: 9200
            protocol: TCP
            targetPort: 9200
          type: NodePort
      tls:
        selfSignedCertificate:
          disabled: true
        certificate: {}
    monitoring:
      logs: {}
      metrics: {}
    nodeSets:
    - config:
        node.data: true
        node.ingest: true
        node.master: true
      count: 3
      name: master
      podTemplate:
        metadata: {}
        spec:
          containers:
          - env:
            - name: ES_JAVA_OPTS
              value: -Xms2048m -Xmx2048m
            name: elasticsearch
            resources:
              limits:
                cpu: "2"
                memory: 4Gi
              requests:
                cpu: "1"
                memory: 4Gi
          initContainers:
          - command:
            - sh
            - -c
            - sysctl -w vm.max_map_count=262144
            name: sysctl
            resources: {}
            securityContext:
              privileged: true
          - command:
            - sh
            - -c
            - |
              bin/elasticsearch-plugin install --batch repository-hdfs
            name: install-plugins
      volumeClaimTemplates:
      - metadata:
          name: elasticsearch-data
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 100Gi
          storageClassName: local-path
        status: {}
    transport:
      service:
        metadata: {}
        spec: {}
      tls:
        certificate: {}
    updateStrategy:
      changeBudget: {}
    version: 7.14.1
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Depoly it.

kubectl apply -f kb_monitor.yam
kubectl describe kibana  -n monitor

But can not init pod.

# kubectl describe kibana  -n monitor
Name:         monitor
Namespace:    monitor
Labels:       <none>
Annotations:  common.k8s.elastic.co/controller-version: 1.7.1
API Version:  kibana.k8s.elastic.co/v1
Kind:         Kibana
Metadata:
  Creation Timestamp:  2021-10-18T11:42:16Z
  Generation:          2
  Managed Fields:
    API Version:  kibana.k8s.elastic.co/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:common.k8s.elastic.co/controller-version:
    Manager:      elastic-operator
    Operation:    Update
    Time:         2021-10-18T11:42:16Z
    API Version:  kibana.k8s.elastic.co/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:associationStatus:
        f:count:
        f:elasticsearchAssociationStatus:
    Manager:      elastic-operator
    Operation:    Update
    Subresource:  status
    Time:         2021-10-18T11:42:16Z
    API Version:  kibana.k8s.elastic.co/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:count:
        f:elasticsearchRef:
          .:
          f:name:
        f:enterpriseSearchRef:
          .:
          f:name:
        f:http:
          .:
          f:service:
            .:
            f:metadata:
            f:spec:
              .:
              f:ports:
                .:
                k:{"port":5601,"protocol":"TCP"}:
                  .:
                  f:name:
                  f:nodePort:
                  f:port:
                  f:protocol:
                  f:targetPort:
              f:type:
          f:tls:
            .:
            f:certificate:
        f:monitoring:
          .:
          f:logs:
          f:metrics:
        f:podTemplate:
          .:
          f:metadata:
          f:spec:
        f:version:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2021-10-18T12:42:04Z
  Resource Version:  4059719
  UID:               da795b91-5373-4d7a-9e5d-f8a8ffa43c45
Spec:
  Count:  1
  Elasticsearch Ref:
    Name:  es-monitor
  Enterprise Search Ref:
    Name:
  Http:
    Service:
      Metadata:
      Spec:
        Ports:
          Name:         https
          Node Port:    30603
          Port:         5601
          Protocol:     TCP
          Target Port:  5601
        Type:           NodePort
    Tls:
      Certificate:
  Monitoring:
    Logs:
    Metrics:
  Pod Template:
    Metadata:
      Creation Timestamp:  <nil>
    Spec:
      Containers:  <nil>
  Version:         7.14.1
Status:
  Association Status:                Pending
  Count:                             0
  Elasticsearch Association Status:  Pending
Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  AssociationError  7m16s (x5 over 67m)  kibana-controller  Association backend for elasticsearch is not configured

Any one can help me, very thanks !

11

Hi @dalei2019,

Disabling the generation of the self-signed certificate also means disabling TLS, which means use HTTP instead of HTTPS (see HTTP configuration | Elastic Cloud on Kubernetes [1.8] | Elastic).

So you need to replace https by http in your manifest:

spec:
  version: 7.14.1
  http:
    service:
      spec:
        ports:
        - name: https <---------- here
          nodePort: 30300
          port: 9200
          protocol: TCP
          targetPort: 9200
        type: NodePort              
    tls:
      selfSignedCertificate:
        disabled: true

Note that requesting the root path / of Elasticsearch doesn't give you a good information about the health of your cluster. Instead, prefer use _cat/health.

1 Like
12

@Thibault_Richard Thanks a lot! It worked !!
After change https to http , kibana started successfully.
Of course, remember to clear the browser cache.