Kibana 8.6.1 keeps Loading Elastic forever when using a JWT token

Elastic Stack Elasticsearch
1

I've been trying to get a JWT token login to work for a few days now. I've made a couple of great steps, I think I've managed to authenticate against the JWT provider (Broadcom IDM). I think I've created a correct role and rolemapping too. I'm getting a response from Kibana too, first error messages but now I'm getting the Elastic logo and then forever the messages "Loading Elastic". I'm wondering if I'm the first to encounter this problem. I've tried to make it work but unfortunately I'm failing and I can't find any hints in the Elastic documentation.

Is there anybody out there who can help me fix the problem?
I've collected (what I believe to be) the relevant configuration items

JWT Token contents
"idp":"Windows","sub":"tuser1","role":["Beheerder_KPV","Beheerder_INFRA"], "iat": 1676033526

Elasticsearch cluster log looks OK (I think):

[2023-02-10T14:38:29,869][INFO ][o.e.x.s.a.j.JwkSetLoader ] [ctbotaels04] Usable PKC: JWKs=[1] algorithms=[RS256] sha256=[9da11876396293244f277e6866db1f0adcff1c42611972318c2e34328fa725d0]
<== Autentication is a success I think

The identity provider is Broadcom IDM, loadbalanced to two Elasticsearch instances. The elasticsearch instances are limited to connect to a single clusternode containing the JWT realm configuration.

realm config in elasticsearch.yml

xpack.security.authc.realms:
  jwt.jwt1:
    order: 0
    client_authentication.type: none
    allowed_issuer: "cibg"
    allowed_audiences: [ "lggng" ]
    allowed_signature_algorithms: [RS256]
    pkc_jwkset_path: jwt/jwtset.json
    claims.principal: sub
    claims.groups: "role"
  native.realm1:
    order: 1
    authentication.enabled: true

GET /_security/role_mapping/beheerders_jwt

{
  "beheerders_jwt": {
    "enabled": true,
    "roles": [
      "beheer"
    ],
    "rules": {
      "all": [
        {
          "field": {
            "realm.name": "jwt1"
          }
        },
        {
          "field": {
            "groups": "Beheerder_INFRA"
          }
        }
      ]
    },
    "metadata": {}
  }
}

GET /_security/role/beheer

{
  "beheer": {
    "cluster": [],
    "indices": [
      {
        "names": [
          "metrics-*",
          "logs*"
        ],
        "privileges": [
          "read",
          "view_index_metadata"
        ],
        "field_security": {
          "grant": [
            "*"
          ],
          "except": []
        },
        "allow_restricted_indices": false
      }
    ],
    "applications": [
      {
        "application": "kibana-.kibana",
        "privileges": [
          "all"
        ],
        "resources": [
          "*"
        ]
      },
      {
        "application": "kibana-.kibana",
        "privileges": [
          "space_all"
        ],
        "resources": [
          "space:default"
        ]
      }
    ],
    "run_as": [],
    "metadata": {},
    "transient_metadata": {
      "enabled": true
    }
  }
}

Kibana.log instance responding, DEBUG level

{
    "client": {
        "ip": "10.136.119.5"
    },
    "http": {
        "request": {
            "method": "GET",
            "mime_type": null,
            "referrer": "",
            "headers": {
                "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
                "accept-encoding": "gzip, deflate, br",
                "accept-language": "nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7",
                "authorization": "[REDACTED]",
                "cache-control": "max-age=0",
                "sec-ch-ua": "\"Google Chrome\";v=\"107\", \"Chromium\";v=\"107\", \"Not=A?Brand\";v=\"24\"",
                "sec-ch-ua-mobile": "?0",
                "sec-ch-ua-platform": "\"Windows\"",
                "sec-fetch-dest": "document",
                "sec-fetch-mode": "navigate",
                "sec-fetch-site": "same-origin",
                "sec-fetch-user": "?1",
                "upgrade-insecure-requests": "1",
                "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36",
                "x-correlationid": "93b5ce95-c7ef-4811-9740-6165138bfa9c",
                "x-forwarded-for": "10.136.108.37,beheer-fix.logging.mso.mhsrijk.nl:8011, vws-dca-wsgo-02.mso.mhsrijk.nl:8080",
                "cookie": "[REDACTED]",
                "host": "log-web-kibana-fix.mso.mhsrijk.nl:443",
                "connection": "Keep-Alive"
            }
        },
        "response": {
            "body": {
                "bytes": 166364
            },
            "status_code": 200,
            "headers": {
                "x-content-type-options": "nosniff",
                "referrer-policy": "no-referrer-when-downgrade",
                "content-security-policy": "script-src 'self' 'unsafe-eval'; worker-src blob: 'self'; style-src 'unsafe-inline' 'self'",
                "kbn-name": "VGA-cluster",
                "kbn-license-sig": "REDACTED",
                "content-type": "text/html; charset=utf-8",
                "cache-control": "private, no-cache, no-store, must-revalidate",
                "vary": "accept-encoding",
                "content-encoding": "gzip"
            },
            "responseTime": 48
        }
    },
    "url": {
        "path": "/app/home",
        "query": ""
    },
    "user_agent": {
        "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"
    },
    "trace": {
        "id": "5d9dae7353b924405b64c4259db13271"
    },
    "service": {
        "node": {
            "roles": ["background_tasks", "ui"]
        }
    },
    "ecs": {
        "version": "8.4.0"
    },
    "@timestamp": "2023-02-10T15:01:16.320+01:00",
    "message": "GET /app/home 200 48ms - 162.5KB",
    "log": {
        "level": "DEBUG",
        "logger": "http.server.response"
    },
    "process": {
        "pid": 130393
    },
    "transaction": {
        "id": "fdfe3d093fb29615"
    }
}

Kibana config

server.port: 5601
server.host: "10.136.121.227"
server.name: "VGA-cluster"
server.ssl.enabled: true
server.ssl.certificate: "/opt/kibana/config/certs/REDACTED.pem"
server.ssl.key: "/opt/kibana/config/certs/REDACTED.key"
elasticsearch.hosts: ["https://10.136.100.102:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "REDACTED"
elasticsearch.ssl.certificate: /opt/kibana-8.6.1/config/certs/REDACTED.cer
elasticsearch.ssl.key: /opt/kibana-8.6.1/config/certs/REDACTED.key
elasticsearch.ssl.certificateAuthorities: [ "/opt/kibana-8.6.1/config/certs/REDACTED.cer" ]
elasticsearch.ssl.verificationMode: full
logging.appenders.default:
  type: file
  fileName: /var/log/kibana/kibana.log
  layout:
    type: json
logging.loggers:
  - name: http.server.response
    level: trace
path.data: /var/data/kibana
pid.file: /var/data/kibana/kibana.pid
xpack.security.encryptionKey: "REDACTED"
xpack.reporting.encryptionKey: "REDACTED"
xpack.encryptedSavedObjects.encryptionKey: "REDACTED"

With kind regards,
Frits

loading

2

Exactly where did you try to use the JWT token? The JWT realm is designed mainly for non-interactive usage, e.g. an application tries to talk to Elasticsearch either on behalf of its user or itself. It is not meant to be used for logging in via Kibana UI.

[2023-02-10T14:38:29,869][INFO ][o.e.x.s.a.j.JwkSetLoader ] [ctbotaels04] Usable PKC: JWKs=[1] algorithms=[RS256] sha256=[9da11876396293244f277e6866db1f0adcff1c42611972318c2e34328fa725d0]

This logging line does not mean authentication is successful. It only means the JWT realm is able to load usable JWK set from jwt/jwtset.json.

I suspect authentication is not successful. I suggest you call Elasticsearch directly with a CLI like curl to see whether authentication is successful by observing the response, e.g.:

curl -H "Authorization: Bearer YOUR_JWT_TOKEN_HERE" http-address-of-your-es-clusrer/_security/_authenticate

The JWT token you shared might be incomplete?

"idp":"Windows","sub":"tuser1","role":["Beheerder_KPV","Beheerder_INFRA"], "iat": 1676033526

It does not have an aud claim which is mandatory.

3

. When I leave out the IAT field in the IDM stub it gives met this message: Authentication to realm jwt1 failed - Realm [jwt1] JWT validation failed for token=[cibg/lggng/tuser1]. (Caused by org.elasticsearch.ElasticsearchSecurityException: missing required date claim [iat]), which implies a missing IAT (correct), but also implies "iss=>cibg and aud=>lggng", so both are set.

I'll find the complete tokenstring and will perform a test with curl as described and share the finding.

From what I see now in chrome debug mode it goes to URL://spaces/enter, then loads a bunch of .js and css files and waits forever after that.

image
image925×92 8.25 KB

image
image1403×484 34.8 KB

4

The error below shows up in the google console, but no errors are visible in the kibana or elasticsearch logging. From te message below I have the impression something is not ok with security somehow.

Refused to apply inline style because it violates the following Content Security Policy directive: "style-src 'self' 'nonce-tJzr4IARD9rKv/M12z38zA=='". Either the 'unsafe-inline' keyword, a hash ('sha256-aqNNdDLnnrDOnTNdkJpYlAxKVJtLt9CtFLklmInuUAE='), or a nonce ('nonce-...') is required to enable inline execution. Note that hashes do not apply to event handlers, style attributes and javascript: navigations unless the 'unsafe-hashes' keyword is present.
5

Have you been able to authenticate with JWT token successfully with curl yet?

The error message you share has nothing to do with the JWT realm itself. It might be related to Kibana. Are you accessing it through a proxy? Are you able to access it with other means of authentication, e.g. username/password?

6

I am able to authenticate with username password with a direct connection to the IP address of the server on port 5601. The authentication falls back to the native username/password.

When I try to connect via a LB with an IDM JWT token it keeps hanging and doesn't fallback to native. I expect today I'm able to give it a try with a token and the curl command and share the ouput of that.

7

The Curl command shows a 401 error. The rotating was caused by by a stylesheet error, so it looks that Kibana and/or Elastic hasn't authorised the user of token somehow? So the real question is probably what did I configure wrong after figuring out how to deal with a JWT token.

No error messages in /var/log/elastcsearch-cluster.log

Simular (later) message in kibana.log

{"client":{"ip":"10.136.119.5"},"http":{"request":{"method":"GET","mime_type":null,"referrer":"","headers":{"accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","accept-encoding":"gzip, deflate, br","accept-language":"nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7","authorization":"[REDACTED]","cache-control":"max-age=0","sec-ch-ua":"\"Google Chrome\";v=\"107\", \"Chromium\";v=\"107\", \"Not=A?Brand\";v=\"24\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Windows\"","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"same-origin","sec-fetch-user":"?1","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36","x-correlationid":"052d0d72-4b83-465a-81e3-76fc3d876bab","x-forwarded-for":"10.136.108.37,beheer-fix.logging.mso.mhsrijk.nl:8011, vws-dca-wsgo-02.mso.mhsrijk.nl:8080","cookie":"[REDACTED]","host":"log-web-kibana-fix.mso.mhsrijk.nl:443","connection":"Keep-Alive"}},"response":{"body":{"bytes":166278},"status_code":200,"headers":{"x-content-type-options":"nosniff","referrer-policy":"no-referrer-when-downgrade","content-security-policy":"script-src 'self' 'unsafe-eval'; worker-src blob: 'self'; style-src 'unsafe-inline' 'self'","kbn-name":"VGA-cluster","kbn-license-sig":"cc7f6f33d5509b9f3224eecf78dad0e805474cc3a0b847cdc9af5486a3c9d3c2","content-type":"text/html; charset=utf-8","cache-control":"private, no-cache, no-store, must-revalidate","vary":"accept-encoding","content-encoding":"gzip"},"responseTime":44}},"url":{"path":"/app/home","query":""},"user_agent":{"original":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"},"trace":{"id":"984876f2c2c9f7d70d855f51ae8727c9"},"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-02-14T12:27:41.115+01:00","message":"GET /app/home 200 44ms - 162.4KB","log":{"level":"DEBUG","logger":"http.server.response"},"process":{"pid":822808},"transaction":{"id":"454534800ff67868"}}


curl -I -k https://10.136.120.227:5601 -H "Authorization:Bearer REDACTED"
HTTP/1.1 401 Unauthorized
x-content-type-options: nosniff
referrer-policy: no-referrer-when-downgrade
content-security-policy: script-src 'self' 'unsafe-eval'; worker-src blob: 'self'; style-src 'unsafe-inline' 'self'
kbn-name: VGA-cluster
kbn-license-sig: REDACTED
refresh: 0;url=/login?msg=UNAUTHENTICATED&next=%2F
content-type: text/html; charset=utf-8
cache-control: private, no-cache, no-store, must-revalidate
content-length: 6
Date: Tue, 14 Feb 2023 10:31:59 GMT
Connection: keep-alive
Keep-Alive: timeout=120
type or paste code here

Result in the browser

    line 286 in kui_light.min.css
            // Since this is an unsafe inline script, this code will not run
            // in browsers that support content security policy(CSP). This is
            // intentional as we check for the existence of __kbnCspNotEnforced__ in
            // bootstrap.

Decoded Token payload

{
  "identity": "vriese@mso.mhsrijk.nl",
  "idp": "Windows",
  "role": [
    "Beheerder_KPV",
    "Beheerder_INFRA"
  ],
  "iat": 1676369766,
  "llt": 1676369766,
  "iss": "cibg",
  "aud": "lggng",
  "exp": 1676371237,
  "nbf": 1676369707,
  "cat": 1676370007,
  "tokenid": "817e5e03-c773-4cb4-be6d-5cac06123af1"
}````
8

The created test role beheer

{
  "beheer": {
    "cluster": [],
    "indices": [
      {
        "names": [
          "metrics-*",
          "logs*"
        ],
        "privileges": [
          "read",
          "view_index_metadata"
        ],
        "field_security": {
          "grant": [
            "*"
          ],
          "except": []
        },
        "allow_restricted_indices": false
      }
    ],
    "applications": [
      {
        "application": "kibana-.kibana",
        "privileges": [
          "all"
        ],
        "resources": [
          "*"
        ]
      }
    ],
    "run_as": [],
    "metadata": {},
    "transient_metadata": {
      "enabled": true
    }
  }
}

The test rolemapping is:

{
  "beheerders_jwt": {
    "enabled": true,
    "roles": [
      "beheer"
    ],
    "rules": {
      "all": [
        {
          "field": {
            "realm.name": "jwt1"
          }
        },
        {
          "field": {
            "groups": "Beheerder_INFRA"
          }
        }
      ]
    },
    "metadata": {}
  }
}

The elasticsearch security configuration is

xpack.security.authc.realms:
  jwt.jwt1:
    order: 0
    #token_type: id_token
    client_authentication.type: none
    allowed_issuer: "cibg"
    allowed_audiences: [ "lggng" ]
    allowed_signature_algorithms: [RS256]
    pkc_jwkset_path: jwt/jwtset.json
    claims.principal: sub
    claims.groups: "role"
  native.realm1:
    order: 1
    authentication.enabled: true

It is quite a lot of small configuration items that somehow have to be in sync. Can you see anything wrong here?

9

You need enable elasticsearch server side debug log to see exactly why your JWT authentication fails.

PUT _cluster/settings
{
  "transient": {
    "logger.org.elasticsearch.xpack.security.authc": "trace"
  }
}

Just based on the JWT token content you shared, it does not have sub claim while your YAML configuration needs it to be the principal claims.principal: sub. However, I suspect you might not have shared the entire content of the JWT. So this might not be the actual issue. As I said, please enable ES server side log and share what it says when the authentication fails.

10

Applied the given debug settings, this gives a whole lot of output :smile:
See below. I can see one error in trace when parsing a token but this seems to relate to user elastic, but also some succesful actions after which it seems to loop. I've put the trace output below from the point where I think the authentication attempt started.

[2023-02-15T13:06:29,262][TRACE][o.e.x.s.a.s.SecondaryAuthenticator] [ctbotaels04] no secondary authentication credentials found (the [es-secondary-authorization] header is [null])
[2023-02-15T13:06:29,262][TRACE][o.e.x.s.a.AuthenticatorChain] [ctbotaels04] Found existing authentication [Authentication[effectiveSubject=Subject{version=8.6.1, user=User[username=elastic,roles=[superuser],fullName=null,email=null,metadata={_reserved=true}], realm={Realm[reserved.reserved] on Node[ctbotaels04]}, type=USER, metadata={}},type=REALM]] in request [transport request action [indices:data/write/bulk]]
[2023-02-15T13:06:29,262][TRACE][o.e.x.s.a.AuthenticatorChain] [ctbotaels04] Found existing authentication [Authentication[effectiveSubject=Subject{version=8.6.1, user=User[username=elastic,roles=[superuser],fullName=null,email=null,metadata={_reserved=true}], realm={Realm[reserved.reserved] on Node[ctbotaels04]}, type=USER, metadata={}},type=REALM]] in request [transport request action [indices:data/write/bulk[s]]]
[2023-02-15T13:06:29,362][TRACE][o.e.x.s.a.s.ServiceAccountToken] [ctbotaels04] parsing token bytes 65794a30655841694f694a4b563151694c434a68624763694f694a53557a49314e694973496e673164434936496b786a5a6b68506331524b56454a4a56334935536e566f636a68534d4774574f56633056534973496d74705a434936496d743262323530595842705a5868304d44457562584e764c6d316f63334a70616d7375626d776966512e65794a705a4841694f694a586157356b6233647a4969776963335669496a6f69646e4a705a584e6c5147317a6279357461484e79615770724c6d357349697769636d39735a53493657794a435a57686c5a584a6b5a584a665331425749697769516d566f5a5756795a47567958306c4f526c4a42496c3073496d6c68644349364d5459334e6a51324d6a63324d53776961584e7a496a6f6959326c695a794973496d46315a434936496d786e5a32356e496977695a586877496a6f784e6a63324e4459304d4445784c434a75596d59694f6a45324e7a59304e6a49304f444573496d4e68644349364d5459334e6a51324d6a63344d537769644739725a5735705a434936496d5578593251354e7a41794c5755304d3249744e446b314e5330354e6a63304c5759794d44646b4d324a695a6a49344e434a392e475a66474861725947706b563532616263435376572d48623848365f363363684b66385a36517953715f71706848345a344543754f654b6170564f513243485f676e4744676b735f334849373458775530576167744334797a70477375325a6c4f68526f6a3931584d396b676149577152546f724a4962494b694b364e79755245574439414e52305a6477667a37644c51457a6b44724a30363063444c5f5a7953304d6559507a656c4c386b76716a6b326d7356306e6a42687a484d585476775467794f366d4659344b6e44674f666e75334371513941544c727669376f70785479352d64784d443378393155744a627a445059466a6370426541797474545a4b6e754836487479697534706e6e787743465337494a34666e7563343651515648505346477256784370484970564169545434415264324c7877744f53576654492d2d77704f30565256576761634377567751387267
[2023-02-15T13:06:29,363][TRACE][o.e.x.s.a.s.ServiceAccountToken] [ctbotaels04] service account token expects the 4 leading bytes to be [0, 1, 0, 1], got [123, 34, 116, 121].
[2023-02-15T13:06:29,363][DEBUG][o.e.x.s.a.TokenService   ] [ctbotaels04] built in token service unable to decode token
java.io.IOException: Illegal base64 character 0x2e
        at java.util.Base64$DecInputStream.read(Base64.java:1159) ~[?:?]
        at org.elasticsearch.common.io.Streams.readFully(Streams.java:146) ~[elasticsearch-8.6.1.jar:?]
        at org.elasticsearch.common.io.stream.InputStreamStreamInput.readBytes(InputStreamStreamInput.java:54) ~[elasticsearch-8.6.1.jar:?] ,
<---- snip   snap---->
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:1589) ~[?:?]
[2023-02-15T13:06:29,366][TRACE][o.e.x.s.a.j.JwtAuthenticationToken] [ctbotaels04] Found allowed principal claim name [sub] with value [vriese@mso.mhsrijk.nl]
[2023-02-15T13:06:29,366][TRACE][o.e.x.s.a.RealmsAuthenticator] [ctbotaels04] Found authentication credentials [org.elasticsearch.xpack.security.authc.jwt.JwtAuthenticationToken] for principal [cibg/lggng/vriese@mso.mhsrijk.nl] in request [rest request uri [/_security/_authenticate]]
[2023-02-15T13:06:29,366][TRACE][o.e.x.s.a.RealmsAuthenticator] [ctbotaels04] Checking token of type [org.elasticsearch.xpack.security.authc.jwt.JwtAuthenticationToken] against [4] realm(s)
[2023-02-15T13:06:29,366][TRACE][o.e.x.s.a.RealmsAuthenticator] [ctbotaels04] Trying to authenticate [cibg/lggng/vriese@mso.mhsrijk.nl] using realm [jwt/jwt1] with token [org.elasticsearch.xpack.security.authc.jwt.JwtAuthenticationToken]
[2023-02-15T13:06:29,366][TRACE][o.e.x.s.a.j.JwtUtil      ] [ctbotaels04] Accepted client. Authentication type [NONE].
[2023-02-15T13:06:29,366][TRACE][o.e.x.s.a.j.JwtRealm     ] [ctbotaels04] Realm [jwt1] client authentication succeeded for token=[cibg/lggng/vriese@mso.mhsrijk.nl].
[2023-02-15T13:06:29,366][TRACE][o.e.x.s.a.j.JwtRealm     ] [ctbotaels04] Realm [jwt1] JWT cache miss token=[cibg/lggng/vriese@mso.mhsrijk.nl] key=[org.elasticsearch.common.bytes.BytesArray@4f16ba5e].
[2023-02-15T13:06:29,371][TRACE][o.e.x.s.a.RealmsAuthenticator] [ctbotaels04] Found authentication credentials [org.elasticsearch.xpack.core.security.authc.support.UsernamePasswordToken] for principal [elastic] in request [rest request uri [/_bulk]]
[2023-02-15T13:06:29,371][TRACE][o.e.x.s.a.RealmsAuthenticator] [ctbotaels04] Checking token of type [org.elasticsearch.xpack.core.security.authc.support.UsernamePasswordToken] against [4] realm(s)
[2023-02-15T13:06:29,371][TRACE][o.e.x.s.a.RealmsAuthenticator] [ctbotaels04] Trying to authenticate [elastic] using realm [reserved/reserved] with token [org.elasticsearch.xpack.core.security.authc.support.UsernamePasswordToken]
[2023-02-15T13:06:29,371][DEBUG][o.e.x.s.a.e.ReservedRealm] [ctbotaels04] realm [reserved] authenticated user [elastic], with roles [[superuser]] (cached)
[2023-02-15T13:06:29,371][DEBUG][o.e.x.s.a.RealmsAuthenticator] [ctbotaels04] Authentication of [elastic] using realm [reserved/reserved] with token [UsernamePasswordToken] was [AuthenticationResult{status=SUCCESS, value=User[username=elastic,roles=[superuser],fullName=null,email=null,metadata={_reserved=true}], message=null, exception=null}]
[2023-02-15T13:06:29,371][TRACE][o.e.x.s.a.AuthenticatorChain] [ctbotaels04] Established authentication [Authentication[effectiveSubject=Subject{version=8.6.1, user=User[username=elastic,roles=[superuser],fullName=null,email=null,metadata={_reserved=true}], realm={Realm[reserved.reserved] on Node[ctbotaels04]}, type=USER, metadata={}},type=REALM]] for request [rest request uri [/_bulk]]
[2023-02-15T13:06:29,371][TRACE][o.e.x.s.a.s.SecondaryAuthenticator] [ctbotaels04] no secondary authentication credentials found (the [es-secondary-authorization] header is [null])
[2023-02-15T13:06:29,372][TRACE][o.e.x.s.a.AuthenticatorChain] [ctbotaels04] Found existing authentication [Authentication[effectiveSubject=Subject{version=8.6.1, user=User[username=elastic,roles=[superuser],fullName=null,email=null,metadata={_reserved=true}], realm={Realm[reserved.reserved] on Node[ctbotaels04]}, type=USER, metadata={}},type=REALM]] in request [transport request action [indices:data/write/bulk]]
[2023-02-15T13:06:29,372][TRACE][o.e.x.s.a.AuthenticatorChain] [ctbotaels04] Found existing authentication [Authentication[effectiveSubject=Subject{version=8.6.1, user=User[username=elastic,roles=[superuser],fullName=null,email=null,metadata={_reserved=true}], realm={Realm[reserved.reserved] on Node[ctbotaels04]}, type=USER, metadata={}},type=REALM]] in request [transport request action [indices:data/write/bulk[s]]]
[2023-02-15T13:06:29,375][DEBUG][o.e.x.s.a.j.JwtAuthenticator] [ctbotaels04] Realm [jwt1] successfully parsed JWT token [cibg/lggng/vriese@mso.mhsrijk.nl] with header [{"x5t":"LcfHOsTJTBIWr9Juhr8R0kV9W4U","kid":"kvontapiext01.mso.mhsrijk.nl","typ":"JWT","alg":"RS256"}] and claimSet [{"sub":"vriese@mso.mhsrijk.nl","aud":"lggng","role":["Beheerder_KPV","Beheerder_INFRA"],"nbf":1676462481,"idp":"Windows","tokenid":"e1cd9702-e43b-4955-9674-f207d3bbf284","cat":1676462781,"iss":"cibg","exp":1676464011,"iat":1676462761}]
[2023-02-15T13:06:29,375][TRACE][o.e.x.s.a.j.JwtValidateUtil] [ctbotaels04] JWKs [1], JWT KID [kvontapiext01.mso.mhsrijk.nl], and JWT Algorithm [RS256] before filters.
[2023-02-15T13:06:29,375][TRACE][o.e.x.s.a.j.JwtValidateUtil] [ctbotaels04] JWKs [1] after KID [kvontapiext01.mso.mhsrijk.nl](|null) filter.
[2023-02-15T13:06:29,375][TRACE][o.e.x.s.a.j.JwtValidateUtil] [ctbotaels04] JWKs [1] after Algorithm [RS256](|null) filter.
[2023-02-15T13:06:29,375][DEBUG][o.e.x.s.a.j.JwtValidateUtil] [ctbotaels04] JWKs [1] after Algorithm [RS256] match filter.
[2023-02-15T13:06:29,375][TRACE][o.e.x.s.a.j.JwtValidateUtil] [ctbotaels04] JWT signature validation succeeded with JWK kty=[RSA], jwtAlg=[RS256], jwtKid=[kvontapiext01.mso.mhsrijk.nl], use=[sig], ops=[null]
[2023-02-15T13:06:29,376][TRACE][o.e.x.s.a.AuthenticatorChain] [ctbotaels04] Found existing authentication [Authentication[effectiveSubject=Subject{version=8.6.1, user=User[username=_xpack_security,roles=[],fullName=null,email=null,metadata={}], realm={Realm[__attach.__attach] on Node[ctbotaels04]}, type=USER, metadata={}},type=INTERNAL]] in request [transport request action [indices:data/read/search]]
[2023-02-15T13:06:29,378][TRACE][o.e.x.s.a.AuthenticatorChain] [ctbotaels04] Found existing authentication [Authentication[effectiveSubject=Subject{version=8.6.1, user=User[username=_xpack_security,roles=[],fullName=null,email=null,metadata={}], realm={Realm[__attach.__attach] on Node[ctbotaels04]}, type=USER, metadata={}},type=INTERNAL]] in request [transport request action [indices:data/read/scroll/clear]]
[2023-02-15T13:06:29,378][TRACE][o.e.x.s.a.s.m.NativeRoleMappingStore] [ctbotaels04] Applying role-mapping [beheeruser] to user-model [{metadata.jwt_claim_iss=cibg, metadata.jwt_claim_tokenid=e1cd9702-e43b-4955-9674-f207d3bbf284, metadata.jwt_claim_role=["Beheerder_KPV","Beheerder_INFRA"], metadata.jwt_claim_cat=1676462781, metadata.jwt_claim_idp=Windows, groups=[Beheerder_KPV, Beheerder_INFRA], metadata.jwt_claim_sub=vriese@mso.mhsrijk.nl, metadata.jwt_claim_aud=[lggng], realm.name=jwt1, username=vriese@mso.mhsrijk.nl}] produced role-names [[kibana_user, beheer]]
[2023-02-15T13:06:29,379][TRACE][o.e.x.s.a.s.m.NativeRoleMappingStore] [ctbotaels04] Applying role-mapping [beheerders_jwt] to user-model [{metadata.jwt_claim_iss=cibg, metadata.jwt_claim_tokenid=e1cd9702-e43b-4955-9674-f207d3bbf284, metadata.jwt_claim_role=["Beheerder_KPV","Beheerder_INFRA"], metadata.jwt_claim_cat=1676462781, metadata.jwt_claim_idp=Windows, groups=[Beheerder_KPV, Beheerder_INFRA], metadata.jwt_claim_sub=vriese@mso.mhsrijk.nl, metadata.jwt_claim_aud=[lggng], realm.name=jwt1, username=vriese@mso.mhsrijk.nl}] produced role-names [[beheer]]
[2023-02-15T13:06:29,379][DEBUG][o.e.x.s.a.s.m.NativeRoleMappingStore] [ctbotaels04] Mapping user [UserData{username:vriese@mso.mhsrijk.nl; dn:null; groups:[Beheerder_KPV, Beheerder_INFRA]; metadata:{jwt_claim_sub=vriese@mso.mhsrijk.nl, jwt_claim_role=["Beheerder_KPV","Beheerder_INFRA"], jwt_claim_idp=Windows, jwt_claim_tokenid=e1cd9702-e43b-4955-9674-f207d3bbf284, jwt_claim_cat=1676462781, jwt_claim_aud=[lggng], jwt_claim_iss=cibg}; realm=jwt1}] to roles [[kibana_user, beheer]]
[2023-02-15T13:06:29,379][DEBUG][o.e.x.s.a.j.JwtRealm     ] [ctbotaels04] Realm [jwt1] roles [kibana_user,beheer] for principal=[vriese@mso.mhsrijk.nl].
[2023-02-15T13:06:29,379][DEBUG][o.e.x.s.a.RealmsAuthenticator] [ctbotaels04] Authentication of [cibg/lggng/vriese@mso.mhsrijk.nl] using realm [jwt/jwt1] with token [JwtAuthenticationToken] was [AuthenticationResult{status=SUCCESS, value=User[username=vriese@mso.mhsrijk.nl,roles=[kibana_user,beheer],fullName=null,email=null,metadata={jwt_claim_sub=vriese@mso.mhsrijk.nl, jwt_claim_role=["Beheerder_KPV","Beheerder_INFRA"], jwt_claim_idp=Windows, jwt_claim_tokenid=e1cd9702-e43b-4955-9674-f207d3bbf284, jwt_claim_cat=1676462781, jwt_claim_aud=[lggng], jwt_claim_iss=cibg}], message=null, exception=null}]
[2023-02-15T13:06:29,379][TRACE][o.e.x.s.a.AuthenticatorChain] [ctbotaels04] Established authentication [Authentication[effectiveSubject=Subject{version=8.6.1, user=User[username=vriese@mso.mhsrijk.nl,roles=[kibana_user,beheer],fullName=null,email=null,metadata={jwt_claim_sub=vriese@mso.mhsrijk.nl, jwt_claim_role=["Beheerder_KPV","Beheerder_INFRA"], jwt_claim_idp=Windows, jwt_claim_tokenid=e1cd9702-e43b-4955-9674-f207d3bbf284, jwt_claim_cat=1676462781, jwt_claim_aud=[lggng], jwt_claim_iss=cibg}], realm={Realm[jwt.jwt1] on Node[ctbotaels04]}, type=USER, metadata={}},type=REALM]] for request [rest request uri [/_security/_authenticate]]
[2023-02-15T13:06:29,380][TRACE][o.e.x.s.a.s.SecondaryAuthenticator] [ctbotaels04] no secondary authentication credentials found (the [es-secondary-authorization] header is [null])
[2023-02-15T13:06:29,380][TRACE][o.e.x.s.a.AuthenticatorChain] [ctbotaels04] Found existing authentication [Authentication[effectiveSubject=Subject{version=8.6.1, user=User[username=vriese@mso.mhsrijk.nl,roles=[kibana_user,beheer],fullName=null,email=null,metadata={jwt_claim_sub=vriese@mso.mhsrijk.nl, jwt_claim_role=["Beheerder_KPV","Beheerder_INFRA"], jwt_claim_idp=Windows, jwt_claim_tokenid=e1cd9702-e43b-4955-9674-f207d3bbf284, jwt_claim_cat=1676462781, jwt_claim_aud=[lggng], jwt_claim_iss=cibg}], realm={Realm[jwt.jwt1] on Node[ctbotaels04]}, type=USER, metadata={}},type=REALM]] in request [transport request action [cluster:admin/xpack/security/user/authenticate]]
[2023-02-15T13:06:29,382][TRACE][o.e.x.s.a.RealmsAuthenticator] [ctbotaels04] Found authentication credentials [org.elasticsearch.xpack.core.security.authc.support.UsernamePasswordToken] for principal [kibana_system] in request [rest request uri [/.kibana_8.6.1/_search?rest_total_hits_as_int=true]]
[2023-02-15T13:06:29,383][TRACE][o.e.x.s.a.RealmsAuthenticator] [ctbotaels04] Checking token of type [org.elasticsearch.xpack.core.security.authc.support.UsernamePasswordToken] against [4] realm(s)
[2023-02-15T13:06:29,383][TRACE][o.e.x.s.a.RealmsAuthenticator] [ctbotaels04] Trying to authenticate [kibana_system] using realm [reserved/reserved] with token [org.elasticsearch.xpack.core.security.authc.support.UsernamePasswordToken]
[2023-02-15T13:06:29,383][DEBUG][o.e.x.s.a.e.ReservedRealm] [ctbotaels04] realm [reserved] authenticated user [kibana_system], with roles [[kibana_system]] (cached)
[2023-02-15T13:06:29,383][DEBUG][o.e.x.s.a.RealmsAuthenticator] [ctbotaels04] Authentication of [kibana_system] using realm [reserved/reserved] with token [UsernamePasswordToken] was [AuthenticationResult{status=SUCCESS, value=User[username=kibana_system,roles=[kibana_system],fullName=null,email=null,metadata={_reserved=true}], message=null, exception=null}]
[2023-02-15T13:06:29,383][TRACE][o.e.x.s.a.AuthenticatorChain] [ctbotaels04] Established authentication [Authentication[effectiveSubject=Subject{version=8.6.1, user=User[username=kibana_system,roles=[kibana_system],fullName=null,email=null,metadata={_reserved=true}], realm={Realm[reserved.reserved] on Node[ctbotaels04]}, type=USER, metadata={}},type=REALM]] for request [rest request uri [/.kibana_8.6.1/_search?rest_total_hits_as_int=true]]
[2023-02-15T13:06:29,383][TRACE][o.e.x.s.a.s.SecondaryAuthenticator] [ctbotaels04] no secondary authentication credentials found (the [es-secondary-authorization] header is [null])
[2023-02-15T13:06:29,383][TRACE][o.e.x.s.a.AuthenticatorChain] [ctbotaels04] Found existing authentication [Authentication[effectiveSubject=Subject{version=8.6.1, user=User[username=kibana_system,roles=[kibana_system],fullName=null,email=null,metadata={_reserved=true}], realm={Realm[reserved.reserved] on Node[ctbotaels04]}, type=USER, metadata={}},type=REALM]] in request [transport request action [indices:data/read/search]]
[2023-02-15T13:06:29,389][TRACE][o.e.x.s.a.s.ServiceAccountToken] [ctbotaels04] parsing token byt




type or paste code here

And repeat from here.

Apart from the strange error message it seems to authenticate Ok and get's the planned roles:
[ctbotaels04] Authentication of [cibg/lggng/vriese@mso.mhsrijk.nl] using realm [jwt/jwt1] with token [JwtAuthenticationToken] was [AuthenticationResult{status=SUCCESS, value=User[username=vriese@mso.mhsrijk.nl,roles=[kibana_user,beheer],fullName=null,email=null,metadata={jwt_claim_sub=vriese@mso.mhsrijk.nl, jwt_claim_role=["Beheerder_KPV","Beheerder_INFRA"], jwt_claim_idp=Windows, jwt_claim_tokenid=e1cd9702-e43b-4955-9674-f207d3bbf284, jwt_claim_cat=1676462781, jwt_claim_aud=[lggng], jwt_claim_iss=cibg}], message=null, exception=null}]

11

Thanks. The logs look right to me. The authentication was successful. This logging line is definitive

[2023-02-15T13:06:29,379][DEBUG][o.e.x.s.a.RealmsAuthenticator] [ctbotaels04] Authentication of [cibg/lggng/vriese@mso.mhsrijk.nl] using realm [jwt/jwt1] with token [JwtAuthenticationToken] was [AuthenticationResult{status=SUCCESS, value=User[username=vriese@mso.mhsrijk.nl,roles=[kibana_user,beheer],fullName=null,email=null,metadata={jwt_claim_sub=vriese@mso.mhsrijk.nl, jwt_claim_role=["Beheerder_KPV","Beheerder_INFRA"], jwt_claim_idp=Windows, jwt_claim_tokenid=e1cd9702-e43b-4955-9674-f207d3bbf284, jwt_claim_cat=1676462781, jwt_claim_aud=[lggng], jwt_claim_iss=cibg}], message=null, exception=null}]

The error you observed was the OAuth2 TokenService trying to decode the JWT and fail because both use the Bearer scheme. This error is benign.

So Elasticsearch side is working correctly. The issue is on the Kibana side. Again, JWT is not meant to be used for KIbana's interactive login. So you might want to reconsider that.

12

Hi Yan_Wang,

Good to see the basics of the JWT authentication and autorisation work out. I understand JWT is not fit for interactive logins, but the interactive part of the login is done elsewhere which is the whole idea of using tokens for authentication/authorisation.

However, I need JWT to work with kibana ánd Elasticsearch. We use broadcom IDM for the interactive part and send the JWT token to applications from there. This is the only way possible for us to combine the multiple authentication/autorisation mechanisms.

One thing I haven't tried yet is to enable JWT on all elasticsearch nodes. This might be a problem for Kibana, although it is configured to query just one specific node. I will give it a try and report back the results.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.