OpenID Connect stand-alone Elastic with Keycloak

Elastic Stack Elasticsearch
1

I would like to have a single node ES (no Kibana involved) authenticate against Keycloak for all requests.

I've followed the OpenID Connect without Kibana documentation to the best of my knowledge. However, this and the openID documentation on ES revolves around a redirect uri and external web app. I'm stuck on this because the intention here is to have, among other things, back-end code authenticate against Keycloak as a requirement for hitting ES (with no web apps involved).

My docker-compose file:

version: "2.2"

services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
      - type: bind
        source: ./elasticsearch.yml
        target: /usr/share/elasticsearch/config/elasticsearch.yml
        read_only: true
      - type: bind
        source: ./certs.json
        target: /usr/share/elasticsearch/config/certs.json
        read_only: true
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01
      # - cluster.initial_master_nodes=es01,es02,es03
      # - discovery.seed_hosts=es02,es03
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

volumes:
  certs:
    driver: local
  esdata01:
    driver: local

.env

# Password for the 'elastic' user (at least 6 characters)
ELASTIC_PASSWORD=password

# Password for the 'kibana_system' user (at least 6 characters)
KIBANA_PASSWORD=password

# Version of Elastic products
STACK_VERSION=8.5.3

# Set the cluster name
CLUSTER_NAME=docker-cluster

# Set to 'basic' or 'trial' to automatically start the 30-day trial
# LICENSE=basic
LICENSE=trial

# Port to expose Elasticsearch HTTP API to the host
ES_PORT=9200
#ES_PORT=127.0.0.1:9200

# Port to expose Kibana to the host
KIBANA_PORT=5601
#KIBANA_PORT=80

# Increase or decrease based on the available host memory (in bytes)
MEM_LIMIT=1073741824

# Project namespace (defaults to the current folder name if not set)
#COMPOSE_PROJECT_NAME=myproject

Elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0

# xpack.security.authc.token.enabled: true
# xpack.security.authc.realms.oidc.oidc1:
#   order: 2
#   rp.client_id: "elasticsearch"
#   rp.response_type: code
#   rp.redirect_uri: "https://127.0.0.1:9200"
#   op.issuer: "http://localhost:8080/auth/realms/Elastic"
#   op.authorization_endpoint: "http://localhost:8080/auth/realms/Elastic/protocol/openid-connect/auth"
#   op.token_endpoint: "http://localhost:8080/auth/realms/Elastic/protocol/openid-connect/token"
#   op.jwkset_path: "certs.json"
#   op.userinfo_endpoint: "http://localhost:8080/auth/realms/Elastic/protocol/openid-connect/userinfo"
#   op.endsession_endpoint: "http://localhost:8080/auth/realms/Elastic/protocol/openid-connect/logout"
#   rp.post_logout_redirect_uri: "https://127.0.0.1:9200/"
#   claims.principal: preferred_username

My keycloak instance is running at http://localhost:8080
with a realm "Elastic" and client "elasticsearch"

So far my process is:

  • run docker-compose up -d

  • Inside the elasticsearch container:

    • Run ./bin/elasticsearch-keystore add xpack.security.authc.realms.oidc.oidc1.rp.client_secret and add the client secret under the Keycloak Realm's Client's Credentials section.

    • Uncomment all the lines in elasticsearch.yml

  • Restart ES container

Then, following the OpenID Connect without Kibana steps, I:

{
	"redirect": "http://localhost:8080/auth/realms/Elastic/protocol/openid-connect/auth?response_type=code&redirect_uri=https%3A%2F%2F127.0.0.1%3A9200&state=Hw-RDYrwwFFUYw_9fGFYWsXyWEReCjlkdIMfKz2uBYY&nonce=jerbpKm5n2i48OGS-z5H47oTXcEL33bSOKSrfRlPQGY&client_id=elasticsearch&scope=openid",
	"state": "Hw-RDYrwwFFUYw_9fGFYWsXyWEReCjlkdIMfKz2uBYY",
	"nonce": "jerbpKm5n2i48OGS-z5H47oTXcEL33bSOKSrfRlPQGY",
	"realm": "oidc1"
}

This is where I am unsure on how to proceed with the Authentication flow. In my elasticsearch.yml I don't know what to put as rp.redirect_uri and rp.post_logout_redirect_uri (my ES endpoint is just a filler/guess) given that there is no other web app involved. I'm sure it makes the redirect url returned invalid right now because keycloak gives a 400 error to it.

2

That is not something that OpenID Connect can do. It is explicitly a protocol for logging into a client that can handle flows, it isn't just a backend for username/password (and can't be used that way).

From the official OpenID Connect page:

It allows Clients to verify the identity of the End-User based on the authentication performed by an Authorization Server, as well as to obtain basic profile information about the End-User in an interoperable and REST-like manner.

If you don't have an OIDC aware client (e.g. a webapp, or a native client), you can't use OIDC. There is nothing in OIDC that allows an API server like Elasticsearch to authenticate REST requests via OIDC.

(OK, technically I lied, it is possible to hack something together via the OIDC implicit flow, but it's an abuse of OIDC, and Elasticsearch doesn't currently support it, nor do we have plans to do so)

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.