401 Unauthorized on Kibana 8.13.2 when accessing canvas API

Elastic Stack Kibana
1

We have to deploy ECK and Elasticsearch 8.13.2 cluster on onpremise VM where K8S is installed.
We have created the esadmin user on Elasticsearch with superuser privileges along with allow_restricted-indices: true

We have deployed Kibana 8.13.2 on another K8S cluster where our application is running and passed elasticsearch host as below. We are using the bitnami helm chart for kibana deployment.

kibana:
  enabled: true
  image:
    registry: <xxxx>.azurecr.io
    repository: <xxxx>/kibana
    tag: 8.13.2-debian-12-r0
  ingress:
    enabled: false  
  elasticsearch:
    hosts:
      - "esadmin:<password>@<ip_address>"
    port: 30200
    security:
      tls:
        enabled: true
        verificationMode: "none"
  configuration:
    server:
      basePath: "/backend-api-gateway/kibana"
      rewriteBasePath: true
  extraConfiguration:
    logging.root.level: debug
    server:
      maxPayload: 10485760      
    monitoring:
      kibana:
        collection:
          enabled: false

We are trying the API requests

**curl -H "Elastic-Api-Version: 1"  -v "http://<kibana_svc>.monitoring.svc.cluster.local:5601/backend-api-gateway/kibana/internal/canvas/workpad/find?name=&perPage=10000"**
*   Trying 10.0.48.220:5601...
* Connected to <kibana_svc>.monitoring.svc.cluster.local (10.0.48.220) port 5601 (#0)
> GET /backend-api-gateway/kibana/internal/canvas/workpad/find?name=&perPage=10000 HTTP/1.1
> Host: <kibana_svc>.monitoring.svc.cluster.local:5601
> User-Agent: curl/7.81.0
> Accept: */*
> Elastic-Api-Version: 1
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< X-Content-Type-Options: nosniff
< Referrer-Policy: strict-origin-when-cross-origin
< Permissions-Policy: camera=(), display-capture=(), fullscreen=(self), geolocation=(), microphone=(), web-share=()
< Cross-Origin-Opener-Policy: same-origin
< Content-Security-Policy: script-src 'report-sample' 'self'; worker-src 'report-sample' 'self' blob:; style-src 'report-sample' 'self' 'unsafe-inline'
< kbn-name: <kibana_svc>-76bbd5df75-4s595
< kbn-license-sig: 02105b5ee33f0e3ae7101d4f01ae86794c83e63bcdd54cffc50c7dc58cba8115
< content-type: application/json; charset=utf-8
< cache-control: private, no-cache, no-store, must-revalidate
< content-length: 66
< Date: Fri, 02 Aug 2024 08:41:55 GMT
< Connection: keep-alive
< Keep-Alive: timeout=120
<
* Connection #0 to host <kibana_svc>.monitoring.svc.cluster.local left intact
{"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}

But we are also trying to pass the Authorization as below

**curl -u esadmin:<password> -H "kbn-xsrf: true" -H "Elastic-Api-Version: 1"  -v "http://<kibana_svc>.monitoring.svc.cluster.local:5601/backend-api-gateway/kibana/internal/canvas/workpad/find?name=&perPage=10000"**
*   Trying 10.0.48.220:5601...
* Connected to <kibana_svc>.monitoring.svc.cluster.local (10.0.48.220) port 5601 (#0)
* Server auth using Basic with user 'esadmin'
> GET /backend-api-gateway/kibana/internal/canvas/workpad/find?name=&perPage=10000 HTTP/1.1
> Host: <kibana_svc>.monitoring.svc.cluster.local:5601
> Authorization: Basic ZXNhZG1pbjpXS2Q5ZDB6WUIxM1U2V2pKWnc4MzAwUDA=
> User-Agent: curl/7.81.0
> Accept: */*
> kbn-xsrf: true
> Elastic-Api-Version: 1
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< elastic-api-version: 1
< x-content-type-options: nosniff
< referrer-policy: strict-origin-when-cross-origin
< permissions-policy: camera=(), display-capture=(), fullscreen=(self), geolocation=(), microphone=(), web-share=()
< cross-origin-opener-policy: same-origin
< content-security-policy: script-src 'report-sample' 'self'; worker-src 'report-sample' 'self' blob:; style-src 'report-sample' 'self' 'unsafe-inline'
< kbn-name: <kibana_svc>-76bbd5df75-4s595
< kbn-license-sig: 02105b5ee33f0e3ae7101d4f01ae86794c83e63bcdd54cffc50c7dc58cba8115
< content-type: application/json; charset=utf-8
< cache-control: private, no-cache, no-store, must-revalidate
< content-length: 25
< accept-ranges: bytes
< Date: Fri, 02 Aug 2024 08:42:31 GMT
< Connection: keep-alive
< Keep-Alive: timeout=120
<
* Connection #0 to host <kibana_svc>.monitoring.svc.cluster.local left intact

Can you please help/suggest, it will be great help for us.

Thank you very much.

2

Below are the logs from kibana, please let me know if more logs are required, only copied the logs with error.

2024-08-02T08:21:31.253103524Z [2024-08-02T08:21:31.252+00:00][DEBUG][http.server.Kibana.cookie-session-storage] Error: Unauthorized
2024-08-02T08:21:31.253247827Z [2024-08-02T08:21:31.253+00:00][DEBUG][plugins.security.basic.basic] Trying to authenticate user request to /status.
2024-08-02T08:21:31.253373429Z [2024-08-02T08:21:31.253+00:00][DEBUG][plugins.security.basic.basic] Redirecting request to Login page.
2024-08-02T08:21:31.255537465Z [2024-08-02T08:21:31.255+00:00][DEBUG][http.server.response] GET /status 302 4ms
2024-08-02T08:21:31.257509399Z [2024-08-02T08:21:31.257+00:00][DEBUG][http.server.Kibana.cookie-session-storage] Error: Unauthorized
2024-08-02T08:21:31.257597700Z [2024-08-02T08:21:31.257+00:00][DEBUG][plugins.security.basic.basic] Trying to authenticate user request to /login?next=%2Fbackend-api-gateway%2Fkibana%2Fstatus.
2024-08-02T08:21:31.257716902Z [2024-08-02T08:21:31.257+00:00][DEBUG][plugins.security.http] Trying to authenticate user request to /login?next=%2Fbackend-api-gateway%2Fkibana%2Fstatus.
2024-08-02T08:21:31.257875005Z [2024-08-02T08:21:31.257+00:00][DEBUG][plugins.security.http] Authorization header is not presented.
2024-08-02T08:21:31.258156810Z [2024-08-02T08:21:31.258+00:00][DEBUG][plugins.security.authentication] Could not handle authentication attempt
2024-08-02T08:21:31.270891726Z [2024-08-02T08:21:31.270+00:00][DEBUG][http.server.response] GET /login?next=%2Fbackend-api-gateway%2Fkibana%2Fstatus 200 14ms - 104.1KB
2024-08-02T08:21:31.441724021Z [2024-08-02T08:21:31.441+00:00][DEBUG][elasticsearch.query.data] 200 - 190.0B