Fleet Server v8.10.3 Security Update

Announcements Security Announcements
1

Fleet Server Insertion of Sensitive Information into Log File (ESA-2023-20)

An issue was discovered in Fleet Server >= v8.10.0 and < v8.10.3 where Agent enrollment tokens are being inserted into the Fleet Server’s log file in plain text.

These enrollment tokens could allow someone to enroll an agent into an agent policy, and potentially use that to retrieve other secrets in the policy including for Elasticsearch and third-party services. Alternatively a threat actor could potentially enroll agents to the clusters and send arbitrary events to Elasticsearch.

Affected Versions:
Fleet Server >= v8.10.0 and < v8.10.3

Solutions and Mitigations:
If an affected version is being utilized then upgrade to Fleet Server v8.10.3 or above. If there are ephemeral containers configured to use a token, ensure they are replaced before deleting and revoking the old token. Delete the existing enrollment tokens and create new ones.

Addendum

If there are reasons why 8.10.0 to 8.10.2 Fleet Server installations cannot be upgraded please see workarounds below:

  1. Standalone Fleet Servers
    To prevent sensitive information being written to the logs on a local Fleet Server machine the log level needs to be changed from info to warning.
    Configure logging for standalone Elastic Agents | Fleet and Elastic Agent Guide [8.10] | Elastic
    elastic-agent.yml
    to
    agent.logging.level: warning
    or in the standalone fleet server policy set it to:
agent:
  logging:
    level: warning
  1. Standalone Fleet Servers alternative
    Users could opt to keep the logs locally, reducing the exposure of secrets to users with access rights to read local logs. Logs can be prevented from being shipping to Elasticsearch by setting:
agent:
  monitoring:
    logs: false

  1. Managed Fleet Servers on-premise
    The logging level can be changed through the UI from info to warning where the default level is set to info. See: Monitor Elastic Agents | Fleet and Elastic Agent Guide [8.10] | Elastic
    or disable shipping logs to Elasticsearch:
    Monitor Elastic Agents | Fleet and Elastic Agent Guide [8.10] | Elastic

  2. In the case of both standalone and managed once the steps above have been followed users should also ensure the log entries are free of sensitive information by deleting those entries from Elasticsearch. First step is to confirm if the sensitive information is within the logs. Within Kibana utilize the following KQL query:

logs-elastic_agent*: event.dataset:"elastic_agent.fleet_server" and agent.version: (8.10.0 OR 8.10.1 OR 8.10.2) and (message:"Found enrollment key" or message:"Checking enrollment key from database") and not message:"[redacted]"

The same can be done in Elasticsearch dev tools ensuring you change the @timestamp range appropriately:

GET /logs-elastic_agent*/_search
{
"sort": [ { "@timestamp": { "order": "desc" } } ],
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "bool": {
            "filter": [
              { "bool": { "should": [ { "term": { "event.dataset": { "value": "elastic_agent.fleet_server" } } } ], "minimum_should_match": 1 } },
              { "bool": { "should": [ { "query_string": { "query": "agent.version:(8.10.0 OR 8.10.1 OR 8.10.2)" } } ], "minimum_should_match": 1 } },
              { "bool": { "should": [
                    { "bool": { "should": [ { "match_phrase": { "message": "Found enrollment key" } } ], "minimum_should_match": 1 } },
                    { "bool": { "should": [ { "match_phrase": { "message": "Checking enrollment key from database" } } ], "minimum_should_match": 1 } }
                  ], "minimum_should_match": 1 }
              },
              { "bool": { "must_not": { "bool": { "should": [ { "match_phrase": { "message": "[redacted]" } } ], "minimum_should_match": 1 } } } }
            ]
          }
        },
        {
          "range": { "@timestamp": { "format": "strict_date_optional_time", "gte": "2023-10-17T00:00:00.000Z", "lte": "2023-10-18T00:00:00.000Z" } }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

If sensitive data is found it can be redacted with the Update query in Kibana Dev tools - please change the @timestamp range appropriately:

POST /logs-elastic_agent*/_update_by_query?pipeline=_none&pretty=true&allow_no_indices=false&wait_for_completion=true&expand_wildcards=all&conflicts=proceed
{
  "script": {
    "source": "if (ctx._source.message != null)  {ctx?._source.message = /Found enrollment key .*/.matcher(ctx._source.message).replaceAll('Found enrollment key [redacted]');}  if (ctx._source.message != null)   {ctx._source.message = /Checking enrollment key from database .*/.matcher(ctx._source.message).replaceAll('Checking enrollment key from database [redacted]');}",
    "lang": "painless"
  },
  "sort": [ { "@timestamp": { "order": "desc" } } ],
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "bool": {
            "filter": [
              { "bool": { "should": [ { "term": { "event.dataset": { "value": "elastic_agent.fleet_server" } } } ], "minimum_should_match": 1 } },
              { "bool": { "should": [ { "query_string": { "query": "agent.version:(8.10.0 OR 8.10.1 OR 8.10.2)" } } ], "minimum_should_match": 1 } },
              { "bool": { "should": [
                    { "bool": { "should": [ { "match_phrase": { "message": "Found enrollment key" } } ], "minimum_should_match": 1 } },
                    { "bool": { "should": [ { "match_phrase": { "message": "Checking enrollment key from database" } } ], "minimum_should_match": 1 } }
                  ], "minimum_should_match": 1 }
              },
              { "bool": { "must_not": { "bool": { "should": [ { "match_phrase": { "message": "[redacted]" } } ], "minimum_should_match": 1 } } } }
            ]
          }
        },
        {
          "range": { "@timestamp": { "format": "strict_date_optional_time", "gte": "2023-10-17T00:00:00.000Z", "lte": "2023-10-18T00:00:00.000Z" } }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

CVSSv3: 8.1 (High) - CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:N
CVE ID: CVE-2023-46667