SAML Authentication: Forbidden access to deployment

Elastic Orchestration Elastic Cloud Enterprise (ECE)
1

Hi,

I have installed Elastic Cloud Enterprise on an AWS virtual machine, and have a ELK stack deployment configured. I have configured an authentication provider (SAML) to login to ECE. It is working correctly and properly assigning roles. All is good there.

Problem is, before having configured with SAML, the admin user would be able to immediately authenticate to a deployment's Kibana without needing to login again. Now, with a SAML provider configured, that does not work for any user from SAML nor the admin user. Instead when launching the Kibana from ECE of the deployment, the screen reads, "Forbidden. You are not allowed to access this resource." Additionally, the logs from ECE have this message for every attempt to open the deployment's Kibana; "Role ece_platform_admin: feature account_activity_readonly not found {}".

Please, any help will be greatly appreciated!

2

Hi Thomas. The automatic SSO into Kibana from the ECE UI is managed by the ECE "security cluster". One thing that helps when troubleshooting these types of issues is to enable TRACE level logs for Elasticsearch SAML and IdP features. In ECE you can go to the "security cluster" deployment's API Console page and set the following via the Elasticsearch API:

PUT /_cluster/settings
{
  "transient": {
    "logger.org.elasticsearch.xpack.security.authc.saml": "TRACE",
    "logger.org.elasticsearch.xpack.idp": "TRACE"
  }
}

From there you should be able to view the logs in the Logging and Metrics cluster (you can get to them from the "Show Dashboard Links > Cluster Logs" link from the security cluster's "Elasticsearch" page).

Can you try the above and see if any SAML IdP errors are logged?

3

Hi @Daniel_Battaglia ,

I enabled the trace logging, and nothing related to SAML appears.

Same as before, only this log message appears for every attempt to launch Kibana:
Role ece_platform_admin: feature account_activity_readonly not found {}

4

Got it, I wonder if the SAML request is not even being initiated. One thing that would be helpful is to see exactly what network request is failing here, and what that response looks like. We can check this using the browser dev tools with a little bit of extra settings. Assuming Chrome:

  • Open the dev tools and switch to the Network tab
  • Open the dev tools settings, scroll down and enable "Auto-open DevTools for popups"
  • Click on the Kibana "Open" link
  • In the new popup, you should see the call to the ECE IdP, something like "$ECE_URL/sso/v1/go/ec:platform:$DEPLOYMENT?acs=..." (if not make sure the "All" or "Doc" filter is selected)

If the URL above is succeeding, then some additional Kibana API or page is likely failing, this should also be visible in the network responses. If you can find the URL returning the Forbidden response, as well as its response body and request/response headers with anything sensitive redacted) then that could be help us figure out whats going on here.

5

Hi @Daniel_Battaglia,

I have found that first call that you indicated. It has a status code of 303.
Right underneath that, there is another call, with the error code forbidden.

Response body

<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><base href="/"><link rel="apple-touch-icon" sizes="180x180" href="favicons/apple-touch-icon-180x180.png"><link rel="icon" type="image/png" sizes="32x32" href="favicons/favicon-32x32.png"><link rel="icon" type="image/png" sizes="16x16" href="favicons/favicon-16x16.png"><link rel="manifest" href="static/manifest.json"><meta name="theme-color" content="#ffffff"><meta name="msapplication-config" content="favicons/ieconfig.xml"><meta name="viewport" content="width=device-width,initial-scale=1"><title>…</title><link href="/vendor/eui_theme_light.min.5b19b0eef530991cad711cd846bae159.css" rel="stylesheet" name="eui-theme" data-theme="light"><link href="/vendor/eui_theme_dark.min.33bd7cf158a3d169f89812dee22c59ab.css" rel="stylesheet" name="eui-theme" data-theme="dark"><link href="/vendor/theme_only_light.3686dabe2778821a3f0b6e976a3df788.css" rel="stylesheet" name="chart-theme" data-theme="light"><link href="/vendor/theme_only_dark.7b62a7f13f4d54fed30928cb8c764dde.css" rel="stylesheet" name="chart-theme" data-theme="dark"><link href="/app.css" rel="stylesheet"></head><body><div id="app-root"></div><script type="text/app-configuration">{"NODE_ENV":"production","CLOUD_UI_APP":"cloud-enterprise-adminconsole","CLOUD_UI_ENV":null,"CLOUD_UI_PRESET":null,"BUILD_TAG":"indeterminate","APP_FAMILY":"cloud-enterprise","APP_PLATFORM":"ece","APP_NAME":"adminconsole","LOG_LEVEL":"info","LOG_DIR":"/app/logs","PORT":5601,"STUNNEL_PORT":5602,"DEDICATED_MASTERS_FIXED_ZONE_COUNT":null,"DEFAULT_REGION":"ece-region","MAX_INSTANCE_COUNT":null,"OAUTH_URL":"api/v1/users/auth/saml/_init","POST_LOGIN_LOCATION":"deployments","TELEMETRY_URL":"https://telemetry.elastic.co","CSP_CONNECT_SOURCES":null,"CLOUD_PRICING_SECRET":null,"CLOUD_USERCONSOLE_APM_SERVER_URL":null,"CLOUD_USERCONSOLE_APM_SERVICE_NAME":null,"CLOUD_USERCONSOLE_APM_SERVICE_PUBLIC_URL":null,"CLOUD_USERCONSOLE_APM_SERVICE_VERSION":null,"CLOUD_USERCONSOLE_RECURLY_PUBLIC_KEY":null,"CLOUD_USERCONSOLE_RUN_FULLSTORY":false,"EXTRA_SUPPORTED_INSTANCE_TYPES":["appsearch","enterprise_search"],"OVERRIDING_INSTANCE_TYPES":null,"DOCUMENTATION_KIND":"ece","FEATURE_ADMIN_CONSOLES":false,"FEATURE_API_KEYS":true,"FEATURE_API_REQUEST_EXAMPLES":true,"FEATURE_BASIC_AUTH_PROXY_PASS":true,"FEATURE_BLOCK_EXPANDING_PLAN_MESSAGES":false,"FEATURE_CLOUD_PORTAL_ENABLED":false,"FEATURE_CONCEAL_PLAN_ERROR_MESSAGE":false,"FEATURE_CONTAINER_SETS":false,"FEATURE_CONVERT_LEGACY_PLANS":true,"FEATURE_CROSS_CLUSTER_SEARCH":true,"FEATURE_DEFAULT_SNAPSHOT_REPOSITORY":false,"FEATURE_DISABLE_EDITING_KIBANA_MEMORY":false,"FEATURE_DISABLE_NODE_CONTROLS_IF_PLAN_PENDING":false,"FEATURE_DISK_QUOTA_OVERRIDE":true,"FEATURE_DISPLAY_FEATURE_CONTROLS":false,"FEATURE_DOWNLOAD_ACTIVITY_JSON":true,"FEATURE_DOWNLOAD_CLUSTER_LOGS":true,"FEATURE_EULA":true,"FEATURE_EXPERIMENTAL_SETTINGS":false,"FEATURE_EXPORT_DEPLOYMENTS":true,"FEATURE_HIDE_ADMIN_REAPPLY_BUTTON":false,"FEATURE_HIDE_CLUSTER_INSTEAD_OF_DELETE":false,"FEATURE_HIDE_CLUSTER_INSTEAD_OF_STOP":false,"FEATURE_HIDE_CONFIG_CHANGE_STRATEGY":false,"FEATURE_HIDE_CREATE_CLUSTER_BUTTON":false,"FEATURE_HIDE_EXTRA_FAILOVER_OPTIONS":false,"FEATURE_HIDE_IRRELEVANT_SECTIONS_FROM_GOV_CLOUD":false,"FEATURE_HIDE_KIBANA_DELETE":false,"FEATURE_HIDE_PAUSE_INSTANCE":false,"FEATURE_HIDE_PLAN_DETAILS":false,"FEATURE_ILM_FEATURE":true,"FEATURE_ILM_MIGRATION":true,"FEATURE_ILM_TEMPLATE_MIGRATION":true,"FEATURE_INCLUDE_FEATURE_FILTER":false,"FEATURE_INSTANCE_CAPACITY_OVERRIDE":true,"FEATURE_INTERCOM_CHAT":false,"FEATURE_IP_FILTERING_ENABLED":false,"FEATURE_LOOKUP_SAAS_USERS":false,"FEATURE_MANAGE_RBAC":true,"FEATURE_MIGRATE_TEMPLATE":false,"FEATURE_MY_COURSES_LINK_IN_PORTAL_TRAINING_TILE":false,"FEATURE_NODE_CONFIGURATIONS":false,"FEATURE_OAUTH":false,"FEATURE_OKTA_AUTHENTICATION_ENABLED":false,"FEATURE_PHONE_HOME":false,"FEATURE_RBAC_PERMISSIONS":true,"FEATURE_READONLY_INDEX_CURATION_TARGETS":false,"FEATURE_REGION_NAMES":false,"FEATURE_REGISTRATION_BUTTONS":false,"FEATURE_RESOURCE_COMMENTS":true,"FEATURE_SAAS_CLUSTER_METRICS":false,"FEATURE_SAAS_FILTERS":false,"FEATURE_SEARCH_CLUSTER_LOCK":false,"FEATURE_SHOW_ADVANCED_EDITOR":true,"FEATURE_SHOW_DASHBOARD_LINKS":false,"FEATURE_SHOW_KIBANA_DETAILS_EVEN_WHEN_HIDDEN":true,"FEATURE_SHOW_NATIVE_MEMORY_PRESSURE":false,"FEATURE_SHOW_SIMPLE_ATTRIBUTION":false,"FEATURE_SHOW_TAKE_SNAPSHOT_BUTTON":true,"FEATURE_SHOW_ACCOUNT_ACTIVITY":true,"FEATURE_SHOW_BILLING_PAGE":true,"FEATURE_ACCOUNT_USAGE_TAB":true,"FEATURE_SHOW_PRICES":false,"FEATURE_SHOW_HELP_PAGE":true,"FEATURE_SHOW_SECURITY_PAGE":true,"FEATURE_ENABLE_INVOICE_ADMIN_ACTIONS":true,"FEATURE_SUDO":false,"FEATURE_TAG_MISMATCH":false,"FEATURE_TEMP_SHIELD_USERS":false,"FEATURE_TOGGLE_CLUSTER_LOCK":false,"FEATURE_TOGGLE_CPU_HARD_LIMIT":true,"FEATURE_TRAFFIC_FILTERING":true,"FEATURE_UC_ILM_BETA_BADGE":false,"FEATURE_WHITELISTING_STACK":false,"FEATURE_CROSS_ENV_CCS_CCR":true,"INTERCOM_URL":"https://elasticcloud-production-chat-us-east-1.s3.amazonaws.com","INTERCOM_DATA":"dates.json","MAILGUN_EVENTS_REGION_ID":null,"MAILGUN_EVENTS_CLUSTER_ID":null,"LOGIN_URL":"api/v1/users/auth/_login","OKTA_URL":"https://auth.elastic.co","CLOUD_STATUS_URL":"https://cloud-status.elastic.co","POLLING_INTERVAL":30000,"MINIMUM_AUTOSCALING_VERSION":"7.11","PORTAL_FEEDS_BASE_PATH":"https://feeds.elastic.co","QR_ISSUER":"Elastic","QR_LABEL":"Elastic Cloud Enterprise","RECOMMENDED_MINIMUM_SYSTEM_DEPLOYMENT_VERSION":"6.6.0","REDUX_LOGGER":false,"ROOT_URL":"api/v0.1","WEBPACK_HOST":"cloud-dev.elastic.co","WEBPACK_PROFILE":false,"OKTA_GOOGLE_CLIENT_ID":null,"OKTA_GOOGLE_IDP":null,"OKTA_AZURE_CLIENT_ID":null,"OKTA_AZURE_IDP":null}</script><script src="/vendor.e55a3974bd1c53ae3a41.js"></script><script src="/app.18d96fd080082e364548.js"></script></body></html>

Response Headers

accept-ranges: bytes
cache-control: public, max-age=0
content-encoding: gzip
content-security-policy: default-src 'none';script-src 'self';worker-src 'self' blob:;connect-src 'self' https://telemetry.elastic.co;img-src 'self' data:;style-src 'self' 'unsafe-inline';manifest-src 'self';font-src 'self';frame-src 'self'
content-type: text/html; charset=UTF-8
date: Mon, 02 Aug 2021 15:56:57 GMT
etag: W/"169c-179d850a098"
last-modified: Fri, 04 Jun 2021 18:37:13 GMT
vary: Accept-Encoding
x-content-type-options: nosniff
x-download-options: noopen
x-frame-options: DENY
x-xss-protection: 0

Request Headers

:authority: $ECE_URL
:method: GET
:path: /errors/sso?error_code=sso.forbidden&sp_login_url=https://$KIBANA_URL&acs=https://KIBANA_URL/api/security/saml/callback&sp_login_url=https://$KIBANA_URL
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9,fr;q=0.8
cookie: EUI_THEME=dark; ec-sso-session-id=V242VEIzc0JqcXdjT3Y3WjFBeG06NkNKbXpBR2RTRjZSTl83TXdJMVNPQQ==; ec-session=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJyZWZyZXNoX3Rva2VuIjoiIiwiYXV0aHpfc291cmNlIjoiWm9va2VlcGVyIiwic3ViIjoiYWRtaW4iLCJuYmYiOjE2Mjc5MTk3MzMsInJvbGVzIjpbImVjZV9wbGF0Zm9ybV9hZG1pbiIsImVjZV9jdXN0b21fZWxldmF0ZWRfYnlfZGVmYXVsdCJdLCJpc3MiOiJmb3VuZC1hZG1pbmNvbnNvbGUiLCJzdWRvIjoxNjU5NDU1NzMzLCJleHAiOjE2Mjc5MjE1MzMsImlhdCI6MTYyNzkxOTczM30.-G8d0yevyW7agIuLxhzdrko1PJULfkVHgHK52F0d_Aw
if-modified-since: Fri, 04 Jun 2021 18:37:13 GMT
if-none-match: W/"169c-179d850a098"
sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"
sec-ch-ua-mobile: ?0
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: same-origin
sec-fetch-user: ?1
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36
6

Thanks, that helps narrow things down. This error indicates the security cluster has rejected the user's SAML request. If you log into the Logging & Metrics cluster's Kibana, are you able to search for the following in the Discover page for the "cluster-logs-*" index pattern?

  • "org.elasticsearch.xpack.idp.saml.idp.SamlIdentityProvider"
  • "org.elasticsearch.xpack.idp.saml.sp.SamlServiceProviderIndex"
  • "org.elasticsearch.xpack.idp.saml.sp.WildcardServiceProviderResolver"

Ideally, one of these searches should return some errors as to why the IdP rejected the request.

Another things to look into is, in the Platform > Settings page, does the "API URL" look correct to you? This should ideally be the same URL you are using to access ECE in the browser. We need this because ECE does not otherwise know about any load balancers or proxies you might have in front of ECE when configured across multiple hosts (and the SAML flow needs this URL).

7

Hi @Daniel_Battaglia,

This did the trick! I decided to look at the API URL of another instance of ECE I have where the SSO was still working, and it was missing the port! Made all the difference! Thanks for all of your help!

8

Ahh fantastic, glad that helped!

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.