Certificate signature failure

[solo1 (solo)]

I tried to follow this (elastic security-basic-setup-https) to configure my elasticsearch and kibana. While the elasticsearch works fine and clients is able to connect successfully with username,password and ca cert(signed with digicert). i'm getting issue connecting to ES from Kibana with warning suggesting "received plaintext http traffic on an https channel" and ": certificate signature failure". wondering if i did something wrong?

My currently docker yaml.

version: '3.8'
services:
   elasticsearch:
        image: XXX:8.9.1-amd64
        container_name: elasticsearch
        restart: unless-stopped
        environment:
        - cluster.name=elasticsearch
        - node.name=db-master
        - bootstrap.memory_lock=true
        - discovery.type=single-node
        - xpack.security.enabled=true
        - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
        - xpack.security.http.ssl.enabled=true
        - xpack.security.http.ssl.keystore.path=XXX/http.p12
        - xpack.security.http.ssl.truststore.path=XXX/http.p12
        - xpack.security.http.ssl.client_authentication=optional
        - xpack.security.transport.ssl.enabled=true
        - xpack.security.transport.ssl.verification_mode=certificate
        - xpack.security.transport.ssl.keystore.path=XXX/http.p12
        - xpack.security.transport.ssl.truststore.path=XXX/http.p12
        ulimits:
            memlock:
                soft: -1
                hard: -1
        mem_limit: 12g
        volumes:
            - esdata:/usr/share/elasticsearch/data
            - XXX:/usr/share/elasticsearch/config/certificates
        ports:
            - 9200:9200
            - 9300:9300
        networks: ['stack']
   kibana:
        image: XXX/kibana:8.9.1-amd64
        environment:
        - SERVERNAME=kibana
        - elasticsearch.username=kibana_system
        - elasticsearch.password=kibanapassword
        - xpack.security.encryptionKey=XXXX
        - elasticsearch.hosts=["https://XXX:9200"]
        - elasticsearch.ssl.certificateAuthorities=certificates/kibana/elasticsearch-ca.pem
        - server.host=0.0.0.0
        - server.ssl.enabled=true
        - server.ssl.key=certificates/xxxx.key
        - server.ssl.certificate=certificates/xxxx.crt
        - xpack.security.enabled=true
        volumes:
            - /XXX/usr/share/kibana/config/certificates
            - kibanadata:/usr/share/kibana/data
        ports: 
            - 5601:5601
        mem_limit: 2g
        networks: ['stack']
        links: ['elasticsearch']
        depends_on: ['elasticsearch']
volumes:
    esdata:
        driver: local
    kibanadata:
        driver: local
networks:
   stack: {}

kibana  | [2023-09-13T16:47:15.546+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
kibana  | [2023-09-13T16:47:34.958+00:00][INFO ][plugins-service] Plugin "cloudChat" is disabled.
kibana  | [2023-09-13T16:47:34.968+00:00][INFO ][plugins-service] Plugin "cloudExperiments" is disabled.
kibana  | [2023-09-13T16:47:34.969+00:00][INFO ][plugins-service] Plugin "cloudFullStory" is disabled.
kibana  | [2023-09-13T16:47:34.969+00:00][INFO ][plugins-service] Plugin "cloudGainsight" is disabled.
kibana  | [2023-09-13T16:47:35.124+00:00][INFO ][plugins-service] Plugin "profiling" is disabled.
kibana  | [2023-09-13T16:47:35.155+00:00][INFO ][plugins-service] Plugin "serverless" is disabled.
kibana  | [2023-09-13T16:47:35.155+00:00][INFO ][plugins-service] Plugin "serverlessObservability" is disabled.
kibana  | [2023-09-13T16:47:35.156+00:00][INFO ][plugins-service] Plugin "serverlessSearch" is disabled.
kibana  | [2023-09-13T16:47:35.156+00:00][INFO ][plugins-service] Plugin "serverlessSecurity" is disabled.
kibana  | [2023-09-13T16:47:35.401+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601
kibana  | [2023-09-13T16:47:35.628+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
kibana  | [2023-09-13T16:47:35.631+00:00][INFO ][preboot] "interactiveSetup" plugin is holding setup: Validating Elasticsearch connection configuration…
kibana  | [2023-09-13T16:47:35.679+00:00][INFO ][root] Holding setup until preboot stage is completed.
kibana  | 
kibana  | 
kibana  | i Kibana has not been configured.
kibana  | 
kibana  | Go to http://0.0.0.0:5601/?code=111111 to get started.
kibana  | 
kibana  |

elasticsearch   | {"@timestamp":"2023-09-13T16:49:41.908Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/172.22.0.2:9200, remoteAddress=/172.22.0.3:48972}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[db-master][transport_worker][T#2]","log.logger":"org.elasticsearch.http.netty4.Netty4HttpServerTransport","elasticsearch.cluster.uuid":"AGzBvKGoT0SQAiLMB2FVTg","elasticsearch.node.id":"oFoS4PWPRdOL9H3hHk6Y4g","elasticsearch.node.name":"db-master","elasticsearch.cluster.name":"elasticsearch"}

[2023-09-13T17:05:21.025+00:00][ERROR][plugins.interactiveSetup.elasticsearch] Failed to authenticate with host "https://XXXX:9200": certificate signature failure