# Kibana 8.11.1 Security Update (ESA-2023-25) **URL:** \ **Category:** Security Announcements\ **Created:** [November 14, 2023, 6:38pm UTC](https://discuss.elastic.co/t/kibana-8-11-1-security-update-esa-2023-25/347149 "2023-11-14T18:38:58Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Levine](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/levine/32/94697_2.png) [@Levine](https://discuss.elastic.co/u/Levine)\ **Post date:** [November 14, 2023, 6:38pm UTC](https://discuss.elastic.co/t/kibana-8-11-1-security-update-esa-2023-25/347149/1 "2023-11-14T18:38:58Z")
# Kibana Insertion of Sensitive Information into Log File (ESA-2023-25) An issue was discovered by Elastic whereby sensitive information may be recorded in Kibana logs in the event of an error. Elastic has released Kibana 8.11.1 which resolves this issue. The error message recorded in the log may contain account credentials for the [kibana\_system](https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-users.html) user, API Keys, and credentials of Kibana end-users. The issue occurs infrequently, only if an error is returned from an Elasticsearch cluster, in cases where there is user interaction and an unhealthy cluster (for example, when returning circuit breaker or no shard exceptions). In Elastic Cloud environments less than 5% of clusters are identified to have been affected. ### Updates Nov 15, 2023 - After additional investigation by Elastic Engineering, it has been determined that Kibana versions 7.x are not affected by this issue. Nov 15, 2023 - Additional details are added for identification and remediation of credentials in logs. Nov 16, 2023 - Added details about the conditions that cause the issue and likelihood of occurrence. Added details on Preventing Ingest of sensitive information from logs and Redacting sensitive information from logs. The CVSS severity rating is revised to 8.0 (High) [CVSS:3.1/AV:A/AC:H/PR:L/UI:N/S:C/C:H/I:H/A:H](https://www.first.org/cvss/calculator/3.1#CVSS:3.1/AV:A/AC:H/PR:L/UI:N/S:C/C:H/I:H/A:H). Dec 1, 2023 - Added suggestion for customers with self-managed monitoring clusters on Elastic Cloud. Dec 8, 2023 - Renamed `kibana_system` to `found-internal-kibana4-server credentials` in Elastic Cloud row about Remediating Sensitive information **Affected Versions** Kibana versions on or after 8.0.0 and before 8.11.1. ## Solutions and Mitigations: The issue is resolved in Kibana 8.11.1 ### Elastic Cloud The following mitigations have been implemented in Elastic Cloud: - We have purged sensitive data that was logged from our monitoring environment. - We have deployed and are currently fortifying a redaction solution so that no new instances of sensitive information are logged in our monitoring environment and in customer’s monitoring clusters. For Elastic Cloud customers **with self-managed monitoring clusters** , affected logs should be reviewed for any potentially sensitive data and if deemed necessary, follow up actions such as purging sensitive data from logs and rotating any potentially exposed credentials should be performed. Elastic Cloud customers **with self-managed monitoring clusters** are also **strongly suggested** to restart all their Kibana instances that are monitored in order to apply additional preventive measures. As additional mitigation, Elastic Cloud customers on affected versions of Kibana are advised to upgrade to 8.11.1. ### Self-Managed Users on affected versions of Kibana in self-managed, ECE, or ECK, should upgrade to Kibana 8.11.1. Affected logs should be reviewed for any potential sensitive data and if deemed necessary, follow up actions such as purging sensitive data from logs and rotating any potentially exposed credentials should be performed. For users that cannot upgrade, see the section [‘Preventing Ingest of Sensitive Information’](#preventing-ingest-of-sensitive-information-in-logs-13) for mitigation actions that can be applied. ## Reviewing Logs for Sensitive Information This section describes how to review logs to identify instances of potentially sensitive information in your logs. #### Elastic Cloud customers with self-managed monitoring clusters Affected log lines can be identified with the use of the following query: `message: ("headers" AND "x-elastic-product-origin" AND "authorization")` against the data views [that contain Kibana logs on the monitoring cluster](https://www.elastic.co/guide/en/cloud/current/ec-enable-logging-and-monitoring.html#ec-access-kibana-monitoring) #### Self-Managed _with_ Elastic Stack Monitoring If you are using [Elastic Stack to ingest Kibana logs](https://www.elastic.co/guide/en/kibana/current/configuring-monitoring.html), affected log lines can be identified with the use of the following query: `message: ("headers" AND "x-elastic-product-origin" AND "authorization")` against the data views that contain Kibana logs on the monitoring cluster: - If [Elastic Agent Collection](https://www.elastic.co/guide/en/kibana/current/monitoring-elastic-agent.html) is configured: `logs-kibana.*-default` - If [Filebeat Collection](https://www.elastic.co/guide/en/kibana/current/monitoring-metricbeat.html) is configured: `filebeat-{version}` , where `{version}` is the installed Kibana version. #### Self-Managed _without_ Elastic Stack Monitoring If you are not ingesting Kibana logs in the Elastic Stack, you can search the log files on disk for occurrences where all the terms `“headers”`, `“x-elastic-product-origin”` and `“authorization”` are present in the same log line. #### ECE On ECE, affected log lines can be identified with the use of the following query: `message: ("headers" AND "x-elastic-product-origin" AND "authorization")` against the `cluster-logs-*` indices in the [logging and metrics cluster.](https://www.elastic.co/guide/en/cloud-enterprise/current/ece-monitoring-ece-access.html) #### ECK If [Stack Monitoring is enabled](https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-stack-monitoring.html), affected log lines can be identified with the use of the following query: `message: ("headers" AND "x-elastic-product-origin" AND "authorization")` against the data views that contain Kibana logs on the monitoring cluster: - If [Elastic Agent Collection](https://www.elastic.co/guide/en/kibana/current/monitoring-elastic-agent.html) is configured: `logs-kibana.*-default` - If [Filebeat Collection](https://www.elastic.co/guide/en/kibana/current/monitoring-metricbeat.html) is configured: `filebeat-{version}` , where `{version}` is the installed Kibana version. ### Remediating Sensitive Information If the review reveals that credentials have been included in the logs, the following remediation actions are recommended. | Installation Type | Remediation Actions | | --- | --- | | Elastic Cloud | `found-internal-kibana4-server credentials` credentials are being rotated by Elastic. End user credentials can be changed using the Management \> Users UI in Kibana or the [Change Password API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-change-password.html). | | Archive Docker RPM/DEB | kibana\_system credentials can be rotated [using one of the available methods](https://www.elastic.co/guide/en/elasticsearch/reference/current/change-passwords-native-users.html) if a native user is used. If service account tokens are used, you need to [delete the logged token](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/security-api-delete-service-token.html) and [create a new token](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/security-api-create-service-token.html) End user credentials can be changed using the Management \> Users UI in Kibana or the [Change Password API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-change-password.html). | | ECE | `found-internal-kibana4-server` credentials can be rotated using advanced editor: 1. Go to advanced editor 2. Find ".resources.kibana[0].plan.cluster\_topology[0].kibana.system\_settings" 3. Set "elasticsearch\_password" to a random string 4. submit End user credentials can be changed using the Management \> Users UI in Kibana or the [Change Password API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-change-password.html). | | ECK | Kibana system user credentials [can be rotated](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-rotate-credentials.html) by either of the following options: - Delete `$KIBANA_NAME-kibana-user` secret in the Kibana K8s namespace and restart the ECK operator. - Delete `$KIBANA_NAME-kibana-user` and `$NAMESPACE-$KIBANA_NAME-kibana-user` secrets in the Kibana K8s namespace End user credentials can be changed using the Management \> Users UI in Kibana or the [Change Password API](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-change-password.html). | ### Preventing Ingest of Sensitive Information in Logs For users that cannot upgrade, the following mitigation actions can be applied to prevent ingestion of sensitive information into an elasticsearch logging cluster. The following mitigations are for customers who have enabled the monitoring features for their clusters as outlined in this [guide](https://www.elastic.co/guide/en/kibana/current/configuring-monitoring.html). If you are using [Elastic Stack to ingest Kibana logs](https://www.elastic.co/guide/en/kibana/current/configuring-monitoring.html), use ECE, or [use ECK with stack monitoring enabled](https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-stack-monitoring.html) , you can use an [Ingest Pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html) to redact sensitive information from being ingested in a logging cluster. 1. Create the remediation pipeline: ```auto PUT _ingest/pipeline/redact-esa–2023-25 { "description": "Prevent Insertion of Sensitive Information into Log File (ESA-2023-25)", "processors": [ { "dot_expander": { "description": "Expand 'event.original'", "field": "event.original" } }, { "set": { "field": "message", "value": "[redacted]", "if": "ctx.message != null && !ctx.message.toLowerCase().contains('[redacted]') && ctx.message.toLowerCase().contains('headers') && ctx.message.toLowerCase().contains('x-elastic-product-origin') && ctx.message.toLowerCase().contains('authorization')" } }, { "set": { "field": "event.original", "value": "[redacted]", "if": "ctx.event?.original != null && !ctx.event.original.toLowerCase().contains('[redacted]') && ctx.event.original.toLowerCase().contains('headers') && ctx.event.original.toLowerCase().contains('x-elastic-product-origin') && ctx.event.original.toLowerCase().contains('authorization')" } } ] } ``` 1. Identify the indexes or datastreams that your Kibana logs are being ingested into. 2. Identify whether these indices have a default pipeline configured by checking for the [`index.default_pipeline`](https://www.elastic.co/guide/en/elasticsearch/reference/8.11/index-modules.html#index-default-pipeline) setting on the indices themselves and the templates. ```auto { "pipeline": { "name": "redact-esa-2023-25" } } ``` The instructions above give API usage examples. Custom pipelines can also be edited using the Ingest Pipelines UI. See [Tutorial: Transform data with custom ingest pipelines](https://www.elastic.co/guide/en/fleet/8.11/data-streams-pipeline-tutorial.html) ### Self-Managed without Elastic Stack Monitoring If you are not ingesting Kibana logs in the Elastic Stack, you should limit access to the directory where Kibana log files are stored. ### ECE logging cluster instructions This assumes the above ingest pipeline `(redact-esa–2023-25)` has been created in the logging cluster as described above. By default the ECE logging cluster ships with a single template for the cluster-logs-\* indices and: - It does not have a `“settings.default_pipeline”` - There are no other templates with `“GET"` You should confirm this remains the case for you (`“GET _template/cluster-logs-*”`, look for `“default_pipeline”`) and (`“GET _template/”`, compare the `“index_patterns”` fields) respectively, and contact support if there are. Otherwise, you will then create an overlapping template just specifying the ingest pipeline: ```auto PUT _template/cluster-logs-esa–2023-25-redaction { "index_patterns" : ["cluster-logs-*"], "order" : 99, "settings": {"default_pipeline": "redact-esa–2023-25"} } ``` The next time the index rolls over, the message field will be replaced by “`[redacted]`” for offending messages. You should then redact sensitive information from already ingested logs as described below. ### Redacting Sensitive Information already ingested in logs The same Ingest Pipeline can be used in conjunction with the [Update By Query API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html) in order to redact already recorded sensitive information in logs. This is applicable for all deployments which includes: Elastic Cloud customers with self-managed monitoring clusters, Self-Managed, ECE, ECK. Repeat for all Kibana logging indices that contain sensitive information. ```auto POST {environment-specific-logging-index}/_update_by_query?pipeline=redact-esa–2023-25&allow_no_indices=false&wait_for_completion=false&expand_wildcards=all&conflicts=proceed { "sort": [ { "@timestamp": { "order": "desc", "unmapped_type": "boolean" } } ], "query": { "bool": { "filter": [ { "bool": { "should": [ { "match_phrase": { "message": "headers" } } ], "minimum_should_match": 1 } }, { "bool": { "should": [ { "match_phrase": { "message": "x-elastic-product-origin" } } ], "minimum_should_match": 1 } }, { "bool": { "should": [ { "match_phrase": { "message": "authorization" } } ], "minimum_should_match": 1 } } ] } } } ``` To check that the update\_by\_query task completes successfully: `GET /_tasks/{taskId from above request}` * * * ### Severity: CVSSv3.1, 8.0 (High) [AV:A/AC:H/PR:L/UI:N/S:C/C:H/I:H/A:H](https://www.first.org/cvss/calculator/3.1#CVSS:3.1/AV:A/AC:H/PR:L/UI:N/S:C/C:H/I:H/A:H) CVE ID: CVE-2023-46671 ---
**Author:** ![Levine](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/levine/32/94697_2.png) [@Levine](https://discuss.elastic.co/u/Levine)\ **Post date:** [November 15, 2023, 11:17pm UTC](https://discuss.elastic.co/t/kibana-8-11-1-security-update-esa-2023-25/347149/2 "2023-11-15T23:17:37Z")
**Updates to ESA-2023-25** Nov 15, 2023 - After additional investigation by Elastic Engineering, it has been determined that Kibana versions 7.x are not affected by this issue. Nov 15, 2023 - Additional details are added for identification and remediation of credentials in logs. ---
**Author:** ![Levine](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/levine/32/94697_2.png) [@Levine](https://discuss.elastic.co/u/Levine)\ **Post date:** [November 17, 2023, 3:18am UTC](https://discuss.elastic.co/t/kibana-8-11-1-security-update-esa-2023-25/347149/3 "2023-11-17T03:18:11Z")
Nov 16, 2023 - Added details about the conditions that cause the issue and likelihood of occurrence. Added details on Preventing Ingest of sensitive information from logs and Redacting sensitive information from logs. The CVSS severity rating is revised to 8.0 (High) [CVSS:3.1/AV:A/AC:H/PR:L/UI:N/S:C/C:H/I:H/A:H](https://www.first.org/cvss/calculator/3.1#CVSS:3.1/AV:A/AC:H/PR:L/UI:N/S:C/C:H/I:H/A:H).