Fleet Server Insertion of Sensitive Information into Log File (ESA-2023-20)
An issue was discovered in Fleet Server >= v8.10.0 and < v8.10.3 where Agent enrollment tokens are being inserted into the Fleet Server’s log file in plain text.
These enrollment tokens could allow someone to enroll an agent into an agent policy, and potentially use that to retrieve other secrets in the policy including for Elasticsearch and third-party services. Alternatively a threat actor could potentially enroll agents to the clusters and send arbitrary events to Elasticsearch.
Affected Versions:
Fleet Server >= v8.10.0 and < v8.10.3
Solutions and Mitigations:
If an affected version is being utilized then upgrade to Fleet Server v8.10.3 or above. If there are ephemeral containers configured to use a token, ensure they are replaced before deleting and revoking the old token. Delete the existing enrollment tokens and create new ones.
Addendum
If there are reasons why 8.10.0 to 8.10.2 Fleet Server installations cannot be upgraded please see workarounds below:
- Standalone Fleet Servers
To prevent sensitive information being written to the logs on a local Fleet Server machine the log level needs to be changed frominfotowarning.
Configure logging for standalone Elastic Agents | Fleet and Elastic Agent Guide [8.10] | Elastic
elastic-agent.yml
to
agent.logging.level: warning
or in the standalone fleet server policy set it to:
agent:
logging:
level: warning
- Standalone Fleet Servers alternative
Users could opt to keep the logs locally, reducing the exposure of secrets to users with access rights to read local logs. Logs can be prevented from being shipping to Elasticsearch by setting:
agent:
monitoring:
logs: false
-
Managed Fleet Servers on-premise
The logging level can be changed through the UI frominfotowarningwhere the default level is set toinfo. See: Monitor Elastic Agents | Fleet and Elastic Agent Guide [8.10] | Elastic
or disable shipping logs to Elasticsearch:
Monitor Elastic Agents | Fleet and Elastic Agent Guide [8.10] | Elastic -
In the case of both standalone and managed once the steps above have been followed users should also ensure the log entries are free of sensitive information by deleting those entries from Elasticsearch. First step is to confirm if the sensitive information is within the logs. Within Kibana utilize the following KQL query:
logs-elastic_agent*: event.dataset:"elastic_agent.fleet_server" and agent.version: (8.10.0 OR 8.10.1 OR 8.10.2) and (message:"Found enrollment key" or message:"Checking enrollment key from database") and not message:"[redacted]"
The same can be done in Elasticsearch dev tools ensuring you change the @timestamp range appropriately:
GET /logs-elastic_agent*/_search
{
"sort": [ { "@timestamp": { "order": "desc" } } ],
"query": {
"bool": {
"must": [],
"filter": [
{
"bool": {
"filter": [
{ "bool": { "should": [ { "term": { "event.dataset": { "value": "elastic_agent.fleet_server" } } } ], "minimum_should_match": 1 } },
{ "bool": { "should": [ { "query_string": { "query": "agent.version:(8.10.0 OR 8.10.1 OR 8.10.2)" } } ], "minimum_should_match": 1 } },
{ "bool": { "should": [
{ "bool": { "should": [ { "match_phrase": { "message": "Found enrollment key" } } ], "minimum_should_match": 1 } },
{ "bool": { "should": [ { "match_phrase": { "message": "Checking enrollment key from database" } } ], "minimum_should_match": 1 } }
], "minimum_should_match": 1 }
},
{ "bool": { "must_not": { "bool": { "should": [ { "match_phrase": { "message": "[redacted]" } } ], "minimum_should_match": 1 } } } }
]
}
},
{
"range": { "@timestamp": { "format": "strict_date_optional_time", "gte": "2023-10-17T00:00:00.000Z", "lte": "2023-10-18T00:00:00.000Z" } }
}
],
"should": [],
"must_not": []
}
}
}
If sensitive data is found it can be redacted with the Update query in Kibana Dev tools - please change the @timestamp range appropriately:
POST /logs-elastic_agent*/_update_by_query?pipeline=_none&pretty=true&allow_no_indices=false&wait_for_completion=true&expand_wildcards=all&conflicts=proceed
{
"script": {
"source": "if (ctx._source.message != null) {ctx?._source.message = /Found enrollment key .*/.matcher(ctx._source.message).replaceAll('Found enrollment key [redacted]');} if (ctx._source.message != null) {ctx._source.message = /Checking enrollment key from database .*/.matcher(ctx._source.message).replaceAll('Checking enrollment key from database [redacted]');}",
"lang": "painless"
},
"sort": [ { "@timestamp": { "order": "desc" } } ],
"query": {
"bool": {
"must": [],
"filter": [
{
"bool": {
"filter": [
{ "bool": { "should": [ { "term": { "event.dataset": { "value": "elastic_agent.fleet_server" } } } ], "minimum_should_match": 1 } },
{ "bool": { "should": [ { "query_string": { "query": "agent.version:(8.10.0 OR 8.10.1 OR 8.10.2)" } } ], "minimum_should_match": 1 } },
{ "bool": { "should": [
{ "bool": { "should": [ { "match_phrase": { "message": "Found enrollment key" } } ], "minimum_should_match": 1 } },
{ "bool": { "should": [ { "match_phrase": { "message": "Checking enrollment key from database" } } ], "minimum_should_match": 1 } }
], "minimum_should_match": 1 }
},
{ "bool": { "must_not": { "bool": { "should": [ { "match_phrase": { "message": "[redacted]" } } ], "minimum_should_match": 1 } } } }
]
}
},
{
"range": { "@timestamp": { "format": "strict_date_optional_time", "gte": "2023-10-17T00:00:00.000Z", "lte": "2023-10-18T00:00:00.000Z" } }
}
],
"should": [],
"must_not": []
}
}
}
CVSSv3: 8.1 (High) - CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:N
CVE ID: CVE-2023-46667