Kibana Insertion of Sensitive Information into Log File (ESA-2023-17)
An issue was discovered by Elastic whereby sensitive information is recorded in Kibana logs in the event of an error. The issue impacts only Kibana version 8.10.0 when logging in the JSON layout or when the pattern layout is configured to log the %meta pattern. Elastic has released Kibana 8.10.1 which resolves this issue. The error object recorded in the log contains request information, which can include sensitive data, such as authentication credentials, cookies, authorization headers, query params, request paths, and other metadata. Some examples of sensitive data which can be included in the logs are account credentials for kibana_system, kibana-metricbeat, or Kibana end-users.
Affected Versions:
Kibana version 8.10.0
Solutions and Mitigations:
The issue is resolved in Kibana 8.10.1. Version 8.10.0 has been removed from our download sites.
Elastic Cloud
Kibana instances of Elastic Cloud Customers on 8.10.0 have been patched to resolve this issue.
Note: If you had upgraded to 8.10.0 and enabled Logging and Monitoring the potential exists that credential material may have been logged in your Logging and Monitoring deployment. We advise you to follow the guidance below to check the Kibana logs for any ingested credentials and perform follow-up actions, such as purging data from logs and rotating any potentially exposed credentials.
The following additional mitigations have been performed by Elastic:
- We have deployed an ingest processor to redact the in-scope fields before they are logged in our monitoring environment.
- We have purged sensitive data that was logged from our monitoring environment before the ingest processor was deployed.
- We are automatically rotating kibana_system and kibana-metricbeat account credentials for all Kibana 8.10.0 deployments in Elastic Cloud
- We reviewed the accesses to our logging environment for the duration of this issue and did not identify any unauthorized activity.
Self-Managed
Users who are running Kibana 8.10.0 self-managed, including ECE or ECK deployments, should upgrade immediately to Kibana 8.10.1.
Users that have run Kibana 8.10.0 instances, should review their logging configuration to determine if they might be affected. The vulnerable logging configurations (defined in kibana.yml) are:
- Using the JSON layout (note that some installation methods log in the JSON layout by default, see table below). For instance:
logging:
appenders:
file:
type: file
fileName: /var/log/kibana/kibana.log
layout:
type: json
- Using the pattern layout with a custom pattern that includes %meta. For instance:
logging:
appenders:
custom_console:
type: console
layout:
type: pattern
pattern: "[%date][%level][%logger] %message %meta"
Review Affected Logs
Affected logs should be reviewed for any potential sensitive data and if deemed necessary, follow up actions such as purging sensitive data from logs and rotating any potentially exposed credentials should be performed.
The following events are of primary interest:
error.meta.meta.request.*error.cause.meta.meta.request.*error.originalError.meta.meta.request.*
More specifically, the following fields are found to contain credentials, Base64 encoded as the value of the Authorization header.
error.meta.meta.request.params.headers.authorization
error.meta.meta.request.options.headers.authorization
error.cause.meta.meta.request.params.headers.authorization
error.cause.meta.meta.request.options.headers.authorization
error.originalError.meta.meta.request.params.headers.authorization
error.originalError.meta.meta.request.options.headers.authorization
Depending on how Kibana is installed and configured, logs can be found in different locations
| Installation Type | Log location |
|---|---|
| All | Systems where you ingest Kibana logs. |
| Archive | Default logging configuration is not affected since it uses the pattern layout without the %meta pattern.By default, Kibana only logs to stdout. If you have explicitly configured Kibana to log to a file, the location of the file is part of the logging configuration in kibana.yml Also note that stdout is also redirected to the Systemd journal |
| Docker | Default logging configuration is affected since it uses the JSON layout. The location of the log file depends on the logging driver in use. By default the json-file https://docs.docker.com/config/containers/logging/json-file driver is used and in Linux based systems the logs of a container are located in /var/lib/docker/containers/<container_id>/<container_id>-json.log |
| RPM/DEB | Default logging configuration is affected since it uses the JSON layout. By default, Kibana logs to /var/log/kibana/kibana.log |
| ECE | Default logging configuration is affected since it uses the JSON layout. Kibana container logs are only persisted when connected to a Logs and metrics cluster. |
| ECK | Default logging configuration is affected since it uses the JSON layout. Kibana container logs are only persisted when connected to a dedicated monitoring cluster. |
If the review reveals that credentials have been included in the logs, the following remediation actions are available
| Installation Type | Available Remediation Actions |
|---|---|
| Archive, Docker, RPM/DEB |
kibana_system credentials can be rotated using one of the available methods if a native user is used. If service account tokens are used, you need to delete the leaked one and create a new token End user credentials can be changed using the Management > Users UI in Kibana or the Change Password API. |
| ECE | found-internal-kibana4-server credentials can be rotated using advanced editor. 1. Go to advanced editor 2. Find " .resources.kibana[0].plan.cluster_topology[0].kibana.system_settings"3. Set " elasticsearch_password" to a complex random string 4. Submit End user credentials can be changed using the Management > Users UI in Kibana or the Change Password API. |
| ECK | Kibana system user credentials can be rotated by either of the following options: - Delete $KIBANA_NAME-kibana-user secret in the Kibana K8s namespace and restart the ECK operator. - Delete $KIBANA_NAME-kibana-userand $NAMESPACE-$KIBANA_NAME-kibana-user secrets in the Kibana K8s namespace End user credentials can be changed using the Management > Users UI in Kibana or the Change Password API. |
Severity Rating
CVSSv3.1: 9.0 (Critical) AV:A/AC:L/PR:L/UI:N/S:C/C:H/I:H/A:H
CVE ID: CVE-2023-31422