Kibana 8.6.1 keeps Loading Elastic forever when using a JWT token

I've been trying to get a JWT token login to work for a few days now. I've made a couple of great steps, I think I've managed to authenticate against the JWT provider (Broadcom IDM). I think I've created a correct role and rolemapping too. I'm getting a response from Kibana too, first error messages but now I'm getting the Elastic logo and then forever the messages "Loading Elastic". I'm wondering if I'm the first to encounter this problem. I've tried to make it work but unfortunately I'm failing and I can't find any hints in the Elastic documentation.

Is there anybody out there who can help me fix the problem?
I've collected (what I believe to be) the relevant configuration items

JWT Token contents
"idp":"Windows","sub":"tuser1","role":["Beheerder_KPV","Beheerder_INFRA"], "iat": 1676033526

Elasticsearch cluster log looks OK (I think):
[2023-02-10T14:38:29,869][INFO ][o.e.x.s.a.j.JwkSetLoader ] [ctbotaels04] Usable PKC: JWKs=[1] algorithms=[RS256] sha256=[9da11876396293244f277e6866db1f0adcff1c42611972318c2e34328fa725d0]
<== Autentication is a success I think

The identity provider is Broadcom IDM, loadbalanced to two Elasticsearch instances. The elasticsearch instances are limited to connect to a single clusternode containing the JWT realm configuration.

realm config in elasticsearch.yml

xpack.security.authc.realms:
  jwt.jwt1:
    order: 0
    client_authentication.type: none
    allowed_issuer: "cibg"
    allowed_audiences: [ "lggng" ]
    allowed_signature_algorithms: [RS256]
    pkc_jwkset_path: jwt/jwtset.json
    claims.principal: sub
    claims.groups: "role"
  native.realm1:
    order: 1
    authentication.enabled: true

GET /_security/role_mapping/beheerders_jwt

{
  "beheerders_jwt": {
    "enabled": true,
    "roles": [
      "beheer"
    ],
    "rules": {
      "all": [
        {
          "field": {
            "realm.name": "jwt1"
          }
        },
        {
          "field": {
            "groups": "Beheerder_INFRA"
          }
        }
      ]
    },
    "metadata": {}
  }
}

GET /_security/role/beheer

{
  "beheer": {
    "cluster": [],
    "indices": [
      {
        "names": [
          "metrics-*",
          "logs*"
        ],
        "privileges": [
          "read",
          "view_index_metadata"
        ],
        "field_security": {
          "grant": [
            "*"
          ],
          "except": []
        },
        "allow_restricted_indices": false
      }
    ],
    "applications": [
      {
        "application": "kibana-.kibana",
        "privileges": [
          "all"
        ],
        "resources": [
          "*"
        ]
      },
      {
        "application": "kibana-.kibana",
        "privileges": [
          "space_all"
        ],
        "resources": [
          "space:default"
        ]
      }
    ],
    "run_as": [],
    "metadata": {},
    "transient_metadata": {
      "enabled": true
    }
  }
}

Kibana.log instance responding, DEBUG level

{
    "client": {
        "ip": "10.136.119.5"
    },
    "http": {
        "request": {
            "method": "GET",
            "mime_type": null,
            "referrer": "",
            "headers": {
                "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
                "accept-encoding": "gzip, deflate, br",
                "accept-language": "nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7",
                "authorization": "[REDACTED]",
                "cache-control": "max-age=0",
                "sec-ch-ua": "\"Google Chrome\";v=\"107\", \"Chromium\";v=\"107\", \"Not=A?Brand\";v=\"24\"",
                "sec-ch-ua-mobile": "?0",
                "sec-ch-ua-platform": "\"Windows\"",
                "sec-fetch-dest": "document",
                "sec-fetch-mode": "navigate",
                "sec-fetch-site": "same-origin",
                "sec-fetch-user": "?1",
                "upgrade-insecure-requests": "1",
                "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36",
                "x-correlationid": "93b5ce95-c7ef-4811-9740-6165138bfa9c",
                "x-forwarded-for": "10.136.108.37,beheer-fix.logging.mso.mhsrijk.nl:8011, vws-dca-wsgo-02.mso.mhsrijk.nl:8080",
                "cookie": "[REDACTED]",
                "host": "log-web-kibana-fix.mso.mhsrijk.nl:443",
                "connection": "Keep-Alive"
            }
        },
        "response": {
            "body": {
                "bytes": 166364
            },
            "status_code": 200,
            "headers": {
                "x-content-type-options": "nosniff",
                "referrer-policy": "no-referrer-when-downgrade",
                "content-security-policy": "script-src 'self' 'unsafe-eval'; worker-src blob: 'self'; style-src 'unsafe-inline' 'self'",
                "kbn-name": "VGA-cluster",
                "kbn-license-sig": "REDACTED",
                "content-type": "text/html; charset=utf-8",
                "cache-control": "private, no-cache, no-store, must-revalidate",
                "vary": "accept-encoding",
                "content-encoding": "gzip"
            },
            "responseTime": 48
        }
    },
    "url": {
        "path": "/app/home",
        "query": ""
    },
    "user_agent": {
        "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"
    },
    "trace": {
        "id": "5d9dae7353b924405b64c4259db13271"
    },
    "service": {
        "node": {
            "roles": ["background_tasks", "ui"]
        }
    },
    "ecs": {
        "version": "8.4.0"
    },
    "@timestamp": "2023-02-10T15:01:16.320+01:00",
    "message": "GET /app/home 200 48ms - 162.5KB",
    "log": {
        "level": "DEBUG",
        "logger": "http.server.response"
    },
    "process": {
        "pid": 130393
    },
    "transaction": {
        "id": "fdfe3d093fb29615"
    }
}

Kibana config

server.port: 5601
server.host: "10.136.121.227"
server.name: "VGA-cluster"
server.ssl.enabled: true
server.ssl.certificate: "/opt/kibana/config/certs/REDACTED.pem"
server.ssl.key: "/opt/kibana/config/certs/REDACTED.key"
elasticsearch.hosts: ["https://10.136.100.102:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "REDACTED"
elasticsearch.ssl.certificate: /opt/kibana-8.6.1/config/certs/REDACTED.cer
elasticsearch.ssl.key: /opt/kibana-8.6.1/config/certs/REDACTED.key
elasticsearch.ssl.certificateAuthorities: [ "/opt/kibana-8.6.1/config/certs/REDACTED.cer" ]
elasticsearch.ssl.verificationMode: full
logging.appenders.default:
  type: file
  fileName: /var/log/kibana/kibana.log
  layout:
    type: json
logging.loggers:
  - name: http.server.response
    level: trace
path.data: /var/data/kibana
pid.file: /var/data/kibana/kibana.pid
xpack.security.encryptionKey: "REDACTED"
xpack.reporting.encryptionKey: "REDACTED"
xpack.encryptedSavedObjects.encryptionKey: "REDACTED"

With kind regards,
Frits

loading

Welcome to our community! :smiley:

Please don't create multiple topics on the same question, it makes it harder for us to help you. Let's continue the discussion in Kibana 8.6.1 keeps Loading Elastic forever when using a JWT token.