Kibana is not able to authenticate elastic search in my k8s deployment

[tauqeerahmad104 (TauqeerAhmad)]

I have to deploy Elasticsearch and kibana on my k8s cluster. The pods were running successfully and worked fine. Then I try adding a username and password for kibana and Elasticsearch. Now Elasticsearch is running smoothly and I can access the API with a username and password. However, the kubana is not able to authenticate the Elasticsearch server. I see Preformatted textthis error.
["warning","process"],"pid":7,"message":"Error [ProductNotSupportedSecurityError]: The client is unable to verify that the server is Elasticsearch due to security privileges on the server side. Some functionality may not be compatible if the server is running an unsupported product.\n

My codes are as follows. For elasticsearch-stateful.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es-cluster
#namespace: logging
spec:
  serviceName: elasticsearch
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:7.16.1
        resources:
            limits:
              cpu: 1000m
            requests:
              cpu: 100m
        ports:
        - containerPort: 9200
          name: rest
          protocol: TCP
        - containerPort: 9300
          name: inter-node
          protocol: TCP
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
        env:
          - name: cluster.name
            value: k8s-logs
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: discovery.seed_hosts
            value: "es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch"
          - name: cluster.initial_master_nodes
            value: "es-cluster-0,es-cluster-1,es-cluster-2"
          - name: ES_JAVA_OPTS
            value: "-Xms512m -Xmx512m"
          - name: ELASTIC_USERNAME
            valueFrom:
              secretKeyRef:
                name: elasticsearch-credentials
                key: username
          - name: ELASTIC_PASSWORD
            valueFrom:
              secretKeyRef:
                name: elasticsearch-credentials
                key: password
          - name: xpack.security.enabled
            value: "true"
      initContainers:
      - name: fix-permissions
        image: busybox
        command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
        securityContext:
          privileged: true
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
      - name: increase-vm-max-map
        image: busybox
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      - name: increase-fd-ulimit
        image: busybox
        command: ["sh", "-c", "ulimit -n 65536"]
        securityContext:
          privileged: true
  volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        app: elasticsearch
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "openebs-hostpath"
      resources:
        requests:
          storage: 3Gi

My kibana file is as follow

apiVersion: apps/v1
kind: Deployment
#namespace: logging
metadata:
  name: kibana
  labels:
    app: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:7.16.1
        resources:
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        env:
          - name: ELASTICSEARCH_URL
            value: http://elastic:admin@elasticsearch:9200
          - name: ELASTIC_USERNAME
            value: elastic
          - name: ELASTIC_PASSWORD
            value: admin
          #- name: ELASTICSEARCH_SSL_VERIFICATIONMODE
            #value: none
          #- name: ELASTIC_USERNAME
            #valueFrom:
              #secretKeyRef:
                #name: elasticsearch-credentials
                #key: username
          #- name: ELASTIC_PASSWORD
            #valueFrom:
              #secretKeyRef:
                #name: elasticsearch-credentials
                #key: password
          - name: xpack.security.enabled
            value: "true"
        ports:
        - containerPort: 5601

When I exec into the Kibana pod and curl the Elastic service with username and password I can see the API.
I am not sure why it sees this error. can you please help me out? Regards,
Tauqeer.