Unable to authenticate Kibana to Elasticsearch on v8.0.1

[Anushree]

I've been attempting to establish authentication between Kibana and Elasticsearch version 8.0.1, using the "kibana_system" user. Unfortunately, I've encountered a 401 error. My Elasticsearch instance is a single-node setup running within a Docker container on a virtual machine (VM). I've configured security for Elasticsearch using PEM format. I'd greatly appreciate any guidance on identifying what might be causing this issue. I've attached my Kibana Dockerfile and docker-compose.yml files for reference.

Error log from kibana container

[DEBUG][elasticsearch.query.data] 401 - 633.0B
2023-09-16T16:08:05.703008000Z GET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip [security_exception]: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]

Error log from Elasticsearch docker container

log.level":"DEBUG", "message":"Authentication of [kibana_system] using realm [reserved/reserved] with token [UsernamePasswordToken] was [AuthenticationResult{status=TERMINATE, value=null, message=failed to authenticate user [kibana_system], exception=null}]", "ecs.version": 

FROM docker.elastic.co/kibana/kibana:8.0.1
USER root
COPY ["CERT_PATH", "/etc/kibana/ssl/"]
RUN apt-get update  && apt-get install openssl wget -y
RUN echo "CERT_PASS" | openssl pkcs12 -in /etc/kibana/ssl/CERT_NAME -out /etc/kibana/ssl/elasticsearch-http-ca.crt -cacerts -nokeys -chain -passin stdin
RUN echo "CERT_PASS" | openssl pkcs12 -in /etc/kibana/ssl/CERT_NAME -clcerts -nokeys -out /etc/kibana/ssl/kb-cert.crt -passin stdin -passout pass:"CERT_PASS"
RUN echo "CERT_PASS" | openssl pkcs12 -in /etc/kibana/ssl/CERT_NAME -nocerts -out /etc/kibana/ssl/kb-key.key -nodes -passin stdin

RUN chown -R kibana:kibana /etc/kibana/ssl/elasticsearch-http-ca.crt && \
chown -R kibana:kibana /etc/kibana/ssl/kb-cert.crt && \
chown -R kibana:kibana /etc/kibana/ssl/kb-key.key

USER kibana

################### Kibana Docker compose #############################################
version: '2'
services:
  kibana:
    image: ACR_Kibana_Image_Path
    network_mode: host
    container_name: kibana
    restart: always
    environment:
      - ELASTICSEARCH_HOSTS="https://127.0.0.1:9200"
      - SERVER_HOST="0.0.0.0"
      - LOGGING_ROOT_LEVEL=debug
      - XPACK_SECURITY_ENABLED=TRUE
      - ELASTICSEARCH_USERNAME=KIBANA_SYSTEM
      - ELASTICSEARCH_PASSWORD="KIBANA_PASWORD"
      - XPACK_SECURITY_ENCRYPTIONKEY="key"
      - XPACK_REPORTING_ENCRYPTIONKEY="key"
      - XPACK_REPORTING_ROLES_ENABLED=FALSE
      - SERVER_SSL_ENABLED=true
      - ELASTICSEARCH_SSL_VERIFICATIONMODE=certificate
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=/etc/kibana/ssl/elasticsearch-http-ca.crt
      - SERVER_SSL_CERTIFICATE=/etc/kibana/ssl/kb-cert.crt
      - SERVER_SSL_KEY=/etc/kibana/ssl/kb-key.key
    ports:
      - 5601:5601
volumes:
  data01:
    driver: local

I also tried to authenticate with elasticsearch service account on docker-compose.yml. However, it is not working either.

 ELASTICSEARCH_SERVICEACCOUNTTOKEN="servicetoken"

Error on kibana container

[DEBUG][elasticsearch.query.data] 401 - 595.0B
2023-09-21T10:03:21.102255000Z GET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip [security_exception]: error attempting to authenticate request

I also see this type of error on both the cases

io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: [.security] org.elasticsearch.index.IndexNotFoundException: no such index [.security]\n\tat