ELK Stack connect to Keycloak via OpenID

devbot · January 1, 2023, 6:53am

I am attempting to have an Elastic Stack (just one node ES and Kibana) authenticate against Keycloak using OpenID.

My docker-compose.yml file:

version: "2.2"

services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
      - type: bind
        source: ./elasticsearch.yml
        target: /usr/share/elasticsearch/config/elasticsearch.yml
        read_only: true
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01
      # - cluster.initial_master_nodes=es01,es02,es03
      # - discovery.seed_hosts=es02,es03
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120


  kibana:
    depends_on:
      es01:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    volumes:
      - certs:/usr/share/kibana/config/certs
      - kibanadata:/usr/share/kibana/data
      - type: bind
        source: ./kibana.yml
        target: /usr/share/kibana/config/kibana.yml
        read_only: true
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
    mem_limit: ${MEM_LIMIT}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  kibanadata:
    driver: local

Elasticsearch.yml:

cluster.name: "docker-cluster"
network.host: 0.0.0.0

# xpack.security.authc.token.enabled: true
# xpack.security.authc.realms.oidc.oidc1:
#   order: 2
#   rp.client_id: "elasticsearch"
#   rp.response_type: code
#   rp.redirect_uri: "https://localhost:5601/api/security/v1/oidc"
#   op.issuer: "http://localhost:8080/auth/realms/Elastic"
#   op.authorization_endpoint: "http://localhost:8080/auth/realms/Elastic/protocol/openid-connect/auth"
#   op.token_endpoint: "http://localhost:8080/auth/realms/Elastic/protocol/openid-connect/token"
#   op.jwkset_path: certs.json
#   op.userinfo_endpoint: "http://localhost:8080/auth/realms/Elastic/protocol/openid-connect/userinfo"
#   op.endsession_endpoint: "http://localhost:8080/auth/realms/Elastic/protocol/openid-connect/logout"
#   rp.post_logout_redirect_uri: "https://localhost:5601/logged_out"
#   claims.principal: preferred_username

kibana.yml

# Default Kibana configuration for docker target
server.host: "0.0.0.0"
server.shutdownTimeout: "5s"
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
monitoring.ui.container.elasticsearch.enabled: true

# xpack.security.authc.providers: [oidc]
# xpack.security.authc.oidc.realm: "oidc1"
# server.xsrf.whitelist: [/api/security/v1/oidc]

# xpack.security.enabled: true

In my local Keycloak instance I have created a realm "Elastic" with client "elasticsearch".

My current process is:

run docker-compose up -d
This successfully sets up the elk stack, with ssl enabled. (this part of docker-compose was modified from the documentation's three node ssl example)
Hit the Start trial API (from my understanding this necessary for xpack's OIDC features)
Inside the elasticsearch container:
- Run ./bin/elasticsearch-keystore add xpack.security.authc.realms.oidc.oidc1.rp.client_secret and add the client secret under the Keycloak Realm's Client's Credentials section.
- Uncomment all the lines in elasticsearch.yml
Restart ES container

ES then crashes and the error I see is:

{
	"@timestamp": "2022-12-31T22:30:11.694Z",
	"log.level": "ERROR",
	"message": "fatal exception while booting Elasticsearch",
	"ecs.version": "1.2.0",
	"service.name": "ES_ECS",
	"event.dataset": "elasticsearch.server",
	"process.thread.name": "main",
	"log.logger": "org.elasticsearch.bootstrap.Elasticsearch",
	"elasticsearch.node.name": "es01",
	"elasticsearch.cluster.name": "docker-cluster",
	"error.type": "java.lang.IllegalStateException",
	"error.message": "security initialization failed",
	"error.stack_trace": "java.lang.IllegalStateException: security initialization failed\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.Security.createComponents(Security.java:577)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.node.Node.lambda$new$16(Node.java:709)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.plugins.PluginsService.lambda$flatMap$0(PluginsService.java:252)\n\tat java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)\n\tat java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)\n\tat java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)\n\tat java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)\n\tat java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)\n\tat java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)\n\tat java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)\n\tat java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)\n\tat java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)\n\tat java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.node.Node.<init>(Node.java:724)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.node.Node.<init>(Node.java:318)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:214)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214)\n\tat org.elasticsearch.server@8.5.3/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)\nCaused by: java.lang.IllegalStateException: Unable to create a IDTokenValidator instance\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectAuthenticator.createIdTokenValidator(OpenIdConnectAuthenticator.java:796)\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectAuthenticator.<init>(OpenIdConnectAuthenticator.java:161)\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectRealm.<init>(OpenIdConnectRealm.java:113)\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.authc.InternalRealms.lambda$getFactories$7(InternalRealms.java:169)\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.authc.Realms.initRealms(Realms.java:288)\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.authc.Realms.<init>(Realms.java:109)\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.Security.createComponents(Security.java:685)\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.Security.createComponents(Security.java:565)\n\t... 17 more\nCaused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config/certs.json\n\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\tat java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)\n\tat java.base/java.nio.file.Files.newByteChannel(Files.java:380)\n\tat java.base/java.nio.file.Files.newByteChannel(Files.java:432)\n\tat java.base/java.nio.file.Files.readAllBytes(Files.java:3287)\n\tat java.base/java.nio.file.Files.readString(Files.java:3365)\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectAuthenticator.readJwkSetFromFile(OpenIdConnectAuthenticator.java:364)\n\tat org.elasticsearch.security@8.5.3/org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectAuthenticator.createIdTokenValidator(OpenIdConnectAuthenticator.java:789)\n\t... 24 more\n"
}

I am unsure how to get past this point or if these steps are even the correct way of going about setting this up.

leandrojmp · January 1, 2023, 1:01pm

You stack trace error has this information:

Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config/certs.json

Did you bind mount this file in your docker compose?

devbot · January 1, 2023, 9:56pm

I've bind mounted the certs.json file (downloaded from Keycloak) now. Repeating the steps above, the elastic container restarts without any error.

But, I can't tell if anything has changed. I am still able to successfully hit https://elastic:password@localhost:9200 in my browser, without being prompted to sign in to the Keycloak instance.

leandrojmp · January 1, 2023, 10:29pm

You are already passing the native user to authenticate, try to pass a user from your keycloak to see if it is working or use your keycloak user in Kibana.

Also, I don't think Elasticsearch won't will prompt you to sign-in, if make an request without any authentication parameters, it will throw an error saying that.

devbot · January 1, 2023, 10:47pm

How do I pass a user from Keycloak? In my keycloak realm I have created a test user.

In my browser I sign in with this user at http://localhost:8080/auth/realms/Elastic/account/#/

If I hit localhost:9200 I am still prompted with a user and password box (which does not accept the keycloak account).

(Also, the Kibana container is not working yet with xpack security enabled. But my end goal is for elasticsearch to authenticate against keycloak itself (for web apps and back end code that use just elastic as a db without kibana involved in the deployment) )

leandrojmp · January 1, 2023, 11:10pm

I'm not sure, I do not use Keycloak.

But from your configuration you have some settings making references to Kibana

here:

rp.redirect_uri: "https://localhost:5601/api/security/v1/oidc"

and here:

rp.post_logout_redirect_uri: "https://localhost:5601/logged_out"

So, You probably need to have Kibana working with Security features to make the authentication works, and this will work for authentication through Kibana, not directly in Elasticsearch.

If you want to configure OpenID to work without Kibana you need to change this documentation.

I do not user OpenID, so I can not help further.

devbot · January 2, 2023, 7:32am

I appreciate your help. Yeah, I'm not sure what the correct configurations would be for an environment without Kibana (or any other web app). I feel like I've gone as far as the OpenID config without Kibana document will take me.

I've created a revised version of my new question(s) here in hopes that someone with OpenID/Keycloak experience can chime in.

leandrojmp · January 2, 2023, 12:12pm

From the documentation on how to configure OpenID without Kibana you have this:

This section describes how a custom web application could use the relevant OpenID Connect REST APIs in order to authenticate the users to Elasticsearch, with OpenID Connect.

So, it takes in consideration that you will use a custom web application.

I have no experience with OpenID, but from the documentation, I'm not sure you can use only elasticsearch.

TimV · January 3, 2023, 3:28am

You shouldn't need any of this manual process.

You can start Elasticsearch with a trial license,
You can bind mount a keystore file just like you bind mount the elasticsearch.yml file.

OpenID Connect is a security protocol for authenticating to a webapp, via a web browser. It is not possible to use OIDC if you don't have a webapp that can handle the authentication flow.

system · January 31, 2023, 3:29am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.