Hi , I'm running the instance of Elastic search and Kibana on a host running under the trail license. The moment i make OIDC changes in Elasticsearch yml for SSO integration, the kibana is not able to read Elasticsearch node. The Elasticsearch node last log line is "es01_1 | {"@timestamp":"2024-05-02T08:04:48.974Z", "log.level": "INFO", "current.health":"GREEN","message":"Cluster health status changed from [RED] to [GREEN] ", but kibana says
" Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 172.25.0.3:9200" . SSO does not work with trail license ? the trail version i have just enabled. I dont see any other logs as well. Can someone show some direction to me ?
Hi @sunilg Welcome to the community.
Yes SSO/SAML work with a trial license
You will need to share your elasticsearch and Kibana .yml
And anything else you did during the configuration.
What version
Thanks for reaching back @stephenb. I did not make any other configuration changes apart from those in yml files mentioned below. The ESK stack version 8.13.2 docker images. ES is deployed as single node cluster with these options in docker-compose file
Elastic node:
- cluster.name=${CLUSTER_NAME}
- cluster.initial_master_nodes=es01
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
Kibana node:
- ELASTICSEARCH_USERNAME=kibana_system
- ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
Here is my Elasticsearch yml :
xpack.security.authc.token.enabled: true
xpack.security.authc.realms.oidc.oidc1:
order: 2
rp.client_id: "workbench-client"
rp.response_type: code
rp.requested_scopes: openid,profile,email,groups
rp.redirect_uri: "https://kibana.com/api/security/oidc/callback"
op.issuer: "https://mydex.hostzone.com"
op.authorization_endpoint: "https://mydex.hostzone.com/auth"
op.token_endpoint: "https://mydex.hostzone.com/token"
op.jwkset_path: "https://mydex.hostzone.com/keys"
op.userinfo_endpoint: "https://mydex.hostzone.com/userinfo"
op.endsession_endpoint: "https://mydex.hostzone.com/auth/logout"
rp.post_logout_redirect_uri: "https://kibana.com/security/logged_out"
claims.principal: email
claims.groups: groups
my kibana.yml goes like this.
xpack.security.authc.providers:
oidc.oidc1:
order: 0
realm: oidc1
basic.basic1:
order: 1
The error message you have indicates Kibana cannot connect to elasticsearch.
Whether that is connectivity or an authentication failure, there's probably some other logs there.
That is a common error message whether people using SSO or not.
Can you share the rest of your compose.
You can also try execing into the Kibana container And trying to curl the elasticsearch from inside the container.
Hi @stephenb , i dont see any error messages. When i replace 8.13.2 docker image with my custom built one i.e elastic_sso:1 ( only replaced yml file with OIDC settings and copied to this location /usr/share/kibana/config/kibana.yml( for kibana) and to this location /usr/share/elasticsearch/config/elasticsearch.yml ( for Elasticsearch)) , i'm getting this error. Which means when i make sso change , the kibana cant seem to connect to Elasticsearch. Apart from WARN and INFO i dont see any errors.
As expected the curl command from kibana container fails , while same curl with localhost on elastic container works fine. I tried to run the get license api.
curl -k -X GET -u elastic:elasticpswd https://localhost:9200/_license
From kibana container
curl -k -X GET https://es01:9200/_license
curl: (7) Failed to connect to es01 port 9200: Connection refused
Kibana last few log lines
kibana_1 | [2024-05-03T14:18:13.090+00:00][WARN ][plugins.actions] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
kibana_1 | [2024-05-03T14:18:13.098+00:00][INFO ][plugins.notifications] Email Service Error: Email connector not specified.
kibana_1 | [2024-05-03T14:18:13.287+00:00][WARN ][plugins.alerting] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
kibana_1 | [2024-05-03T14:18:13.288+00:00][INFO ][plugins.alerting] using indexes and aliases for persisting alerts
kibana_1 | [2024-05-03T14:18:14.414+00:00][WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
kibana_1 | [2024-05-03T14:18:14.962+00:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
kibana_1 | [2024-05-03T14:18:17.122+00:00][INFO ][plugins.securitySolution.endpoint:user-artifact-packager:1.0.0] Registering endpoint:user-artifact-packager task with timeout of [20m], interval of [60s] and policy update batch size of [25]
kibana_1 | [2024-05-03T14:18:17.457+00:00][INFO ][plugins.assetManager] Server is NOT enabled
kibana_1 | [2024-05-03T14:18:17.792+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 172.26.0.3:9200
kibana_1 | [2024-05-03T14:18:18.430+00:00][INFO ][plugins.screenshotting.chromium] Browser executable: /usr/share/kibana/node_modules/@kbn/screenshotting-plugin/chromium/headless_shell-linux_x64/headless_shell
Elastic last 2 log lines:
es01_1 | {"@timestamp":"2024-05-03T14:17:56.579Z", "log.level": "INFO", "message":"Node [{es01}{KwCfSphlRB6KtIxkOoFDMg}] is selected as the current health node.", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][management][T#2]","log.logger":"org.elasticsearch.health.node.selection.HealthNodeTaskExecutor","elasticsearch.cluster.uuid":"RfJJDna2Rja6vq2tP1VC4w","elasticsearch.node.id":"KwCfSphlRB6KtIxkOoFDMg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"new-docker-cluster"}
es01_1 | {"@timestamp":"2024-05-03T14:17:57.407Z", "log.level": "INFO", "current.health":"GREEN","message":"Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.ds-.kibana-event-log-ds-2024.05.02-000001][0]]]).","previous.health":"RED","reason":"shards started [[.ds-.kibana-event-log-ds-2024.05.02-000001][0]]" , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es01][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.AllocationService","elasticsearch.cluster.uuid":"RfJJDna2Rja6vq2tP1VC4w","elasticsearch.node.id":"KwCfSphlRB6KtIxkOoFDMg","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"new-docker-cluster"}
my whole docker compose file
version: "2.2"
services:
setup:
#image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
image: elastic_sso:1
volumes:
- certs:/usr/share/elasticsearch/config/certs
user: "0"
command: >
bash -c '
if [ x${ELASTIC_PASSWORD} == x ]; then
echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
exit 1;
elif [ x${KIBANA_PASSWORD} == x ]; then
echo "Set the KIBANA_PASSWORD environment variable in the .env file";
exit 1;
fi;
if [ ! -f certs/ca.zip ]; then
echo "Creating CA";
bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
unzip config/certs/ca.zip -d config/certs;
fi;
if [ ! -f certs/certs.zip ]; then
echo "Creating certs";
echo -ne \
"instances:\n"\
" - name: es01\n"\
" dns:\n"\
" - es01\n"\
" - localhost\n"\
" ip:\n"\
" - 127.0.0.1\n"\
> config/certs/instances.yml;
bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
unzip config/certs/certs.zip -d config/certs;
fi;
echo "Setting file permissions"
chown -R root:root config/certs;
find . -type d -exec chmod 750 \{\} \;;
find . -type f -exec chmod 640 \{\} \;;
echo "Waiting for Elasticsearch availability";
until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
echo "Setting kibana_system password";
until curl -s -X POST --cacert config/certs/ca/ca.crt -u elastic:${ELASTIC_PASSWORD} -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
echo "All done!";
'
healthcheck:
test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
interval: 1s
timeout: 5s
retries: 120
es01:
depends_on:
setup:
condition: service_healthy
#image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
image: elastic_sso:1
volumes:
- certs:/usr/share/elasticsearch/config/certs
- esdata01:/usr/share/elasticsearch/data
ports:
- ${ES_PORT}:9200
environment:
- node.name=es01
- cluster.name=${CLUSTER_NAME}
- cluster.initial_master_nodes=es01
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
- bootstrap.memory_lock=true
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=certs/es01/es01.key
- xpack.security.http.ssl.certificate=certs/es01/es01.crt
- xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.http.ssl.verification_mode=certificate
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=certs/es01/es01.key
- xpack.security.transport.ssl.certificate=certs/es01/es01.crt
- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.license.self_generated.type=${LICENSE}
mem_limit: ${MEM_LIMIT}
ulimits:
memlock:
soft: -1
hard: -1
healthcheck:
test:
[
"CMD-SHELL",
"curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
]
interval: 10s
timeout: 10s
retries: 120
kibana:
depends_on:
es01:
condition: service_healthy
#image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
image: kibana_sso:1
volumes:
- certs:/usr/share/kibana/config/certs
- kibanadata:/usr/share/kibana/data
ports:
- ${KIBANA_PORT}:5601
environment:
- SERVERNAME=kibana
- ELASTICSEARCH_HOSTS=https://es01:9200
- ELASTICSEARCH_USERNAME=kibana_system
- ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
- ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
mem_limit: ${MEM_LIMIT}
healthcheck:
test:
[
"CMD-SHELL",
"curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
]
interval: 10s
timeout: 10s
retries: 120
volumes:
certs:
driver: local
esdata01:
driver: local
kibanadata:
driver: local
Clearly Kibana cannot connect with elastic....
There is a connectivity/ network issue of some sort.
Looks to me that perhaps your custom image may be an issue.
Not sure what you did there...
Perhaps you should set
network.host: 0.0.0.0
Hi @stephenb, Does SSO/SAML only works with paid versions or is it possible with community edition too?
Hi @Suresh2, SSO is not a part of community edition. You have to upgrade. For more info you can check this page.
@stephenb , thanks tip on configuration around network host. However, adding that in compose file didnt work. Had to add set it up configuration file(yml) file. I think we close this thread/ticket as now its working fine with OIDC provider. Thanks Again
1 Like