Unable to authenticate using custom realm example code security-example-spi-extension

I've installed the provided security-example-spi-extension plugin (elasticsearch/x-pack/qa/security-example-spi-extension at 8.12 · elastic/elasticsearch · GitHub) and when I attempt to authenticate, it goes through CustomAuthenticationFailureHandler code, but never hits the CustomRealm code's supports(), token(), or authenticate(), even though the class is getting constructed and the plugin is getting loaded.

I have created a zip file that includes the jar of the built plugin and the plugin-descriptor.properties file at the root level of the zip. The plugin-descriptor looks like:

name=my plugin

My elasticsearch.yml looks like:

      # Not sure if this name is arbitrary but I've also tried calling it custom (...realms.custom.custom...) here
        order: 0
        username: "test_user"
        filtered_setting: "should be filtered"
    file.esusers.order: 1
    native.native.order: 2
    # Have also tried adding the following which is in the security-example-spi-extension plugin's test code, but also failed in the same way
    #custom_role_mapping.role_map.order: 3

I've created a keystore and have added the password to it like:

echo "secret_password" | ${ES_HOME}/bin/elasticsearch-keystore add --stdin xpack.security.authc.realms.custom.my_realm.password

When trying the following curl commands (latter uses b64 encoded test_user:secret_password as the basic token):

curl --user "test_user:secret_password" localhost:9200/_cluster/health?pretty
curl -H "Authorization: Basic dGVzdF91c2VyOnNlY3JldF9wYXNzd29yZA==" localhost:9200/_cluster/health?pretty

I get:

  "error" : {
    "root_cause" : [
        "type" : "security_exception",
        "reason" : "unable to authenticate user [test_user] for REST request [/_cluster/health?pretty]",
        "header" : {
          "WWW-Authenticate" : "custom-challenge-jubui-failed-authentication-1"
    "type" : "security_exception",
    "reason" : "unable to authenticate user [test_user] for REST request [/_cluster/health?pretty]",
    "header" : {
      "WWW-Authenticate" : "custom-challenge-jubui-failed-authentication-1"
  "status" : 401

Other notes:
-in the CustomRealm constructor I've verified that the username/password comes from my configured values test_user and secret_password and not the defaults.
-I can't seem to get System.out.println to put anything to the log, so I've manually sent logging output to a custom file. Nothing other method in CustomRealm gets hit other than the constructor
-I've modified the CustomAuthenticationFailureHandler so that it displays a custom error and I see the custom error displayed so I know that my plugin is being added and the custom failure handler of the plugin is used
-Elasticsearch startup does show that my plugin was loaded
-No errors/warnings in the elasticsearch log
-I've tried modifying CustomRealm so that it always returns a token/successful authentication response and it always says it supports the request. Of course, these methods don't seem to be hit at all so this fails.

Please help! TIA!

I found that I needed this line in my elasticsearch.yml which wasn't mentioned in Integrating with other authentication systems | Elasticsearch Guide [8.12] | Elastic

xpack.license.self_generated.type: "trial"

If the problem was due to your license, then the logs should have indicated that the custom realm was disabled.

Using a case insensitive search I find no relevant logging for "custom", "realm", "security", or "plugin". I found no relevant output searching for "warn", "error", and "failure" either.

I'll admit that I have no idea if my log is improperly configured

Furthermore, as I said, my authentication failure handler code was still hit even though the custom realm was disabled? This is super strange to me.

Lastly, if it could save anyone time, I would've loved for the licensing issue to have been stated in the docs on the custom realms page Integrating with other authentication systems | Elasticsearch Guide [8.12] | Elastic


Our goal is to make these things obvious in product rather than rely on putting more things in the docs.

You should get this message on startup:

[2024-03-13T12:22:44,224][WARN ][o.e.x.s.a.Realms         ] [node-name] The [type.realm_name] realm has been automatically disabled due to a change in license [active basic license]

and then this message on every failed authentication:

[2024-03-13T12:24:00,783][WARN ][o.e.x.s.a.RealmsAuthenticator] [node-name] Authentication failed using realms [reserved/reserved,file/file1,native/native1]. Realms [type/realm_name] were skipped because they are not permitted on the current license

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.