Unhealthy EDOT/OTEL Collector in Agent 9.4.x

Elastic Stack Elastic Agent
1

Environment:

  • Elastic Agent / EDOT: 9.4.4 (build baed89504b1a1c0a7642d60dc0b6c1b0eb34f100, elastic-otel-collector service.version 9.4.4)
  • OS: Ubuntu 24.04 (HP thin client)
  • Fleet-managed, policy "Thin Client"
  • Agent runtime config:
    agent.internal.runtime.metricbeat.default: otel # metricbeat hosted inside the OTEL collector
    agent.monitoring._runtime_experimental: otel
    agent.grpc.port: 6789 # agent <-> collector mgmt channel

Symptom:
After the reimage + upgrade to 9.4.4, the agent reports DEGRADED (state 3) with:

collector:
error: failed to connect to collector
status: 3
timestamp: "2026-07-28T12:34:44Z"
...
fleet_message: Connected # Fleet link is fine
fleet_state: 2
message: 1 or more components/units in a degraded state
state: 3 # agent overall DEGRADED

The two components that never come up are exactly the two system/metrics-type inputs that run metricbeat inside the EDOT collector (the metricbeatreceiver):

  • system/metrics-default > metricbeatreceiver/_agent-component/system/metrics-default > elasticsearch/_agent-component/default
  • system/metrics-monitoring > metricbeatreceiver/_agent-component/system/metrics-monitoring > elasticsearch/_agent-component/monitoring

Every other input type (log-default, audit/auditd-default, audit/file_integrity-default, synthetics/tcp-default, osquery-default, endpoint, beat/metrics-monitoring, http/metrics-monitoring,
filestream-monitoring) checks in Healthy.

What I've verified (evidence)

  1. Policy vs. running components. Comparing the agent's expected vs. actual component list, the only two missing are the two system/metrics inputs:

Expected (11): log-default, system/metrics-default, audit/auditd-default,
audit/file_integrity-default, synthetics/tcp-default, endpoint,
osquery-default, http/metrics-monitoring, system/metrics-monitoring,
filestream-monitoring, beat/metrics-monitoring

Actual (9): ...minus system/metrics-default and system/metrics-monitoring

All components declare input_spec.binary_name: elastic-otel-collector (EDOT hosts everything).

  1. The collector itself is healthy. collector.log is entirely info-level (3,344 lines, zero warn/error). Both system/metrics metricbeatreceivers are running and emitting "Non-zero metrics"
    every 30s:

"otelcol.component.id":"metricbeatreceiver/_agent-component/system/metrics-default" (1,408 lines)
"otelcol.component.id":"metricbeatreceiver/_agent-component/system/metrics-monitoring" (1,408 lines)
"metricbeat":{"system":{"process":{"events":1,"success":1}}}

So the metricbeat-via-OTEL receivers work at the collector level — the failure is purely on the agent-collector management plane (the agent can't register/manage those two components,
which is why they're dropped from state.yaml/components-actual.yaml even though the collector keeps them alive).

  1. The only non-telemetry lines in elastic-agent.log are these two warnings from internal/pkg/otel/manager/diagnostics.go:137:
  • component ID "system/metrics-default" contains '/', its EDOT diagnostics will be missing from the archive
  • component ID "system/metrics-monitoring" contains '/', its EDOT diagnostics will be missing from the archive

The agent's OTel manager is flagging exactly these two component IDs as mishandled because they contain a / (the system/metrics input type embeds a slash). Notably, sibling IDs that also contain a slash (audit/file_integrity-default, beat/metrics-monitoring) come up healthy — the differentiator is that the two failing ones are the system/metrics metricbeat-via-OTEL components.

  1. This is a regression vs. the previous build. On the prior Agent 9.4.2 (same host, before reimage) the system/metrics-default / system/metrics-monitoring metricbeatreceivers worked (confirmed in older journald telemetry). The break appeared with the Ubuntu 24.04 reimage + Agent 9.4.4 upgrade.

  2. Not the cause (ruled out):

  • Fleet connectivity: healthy (Connected, fleet_state 2).
  • ES exporter auth: works (other components ack events).
  • metrics.log errors: only two "Exporting failed. Rejecting data" with "error":{"message":"context canceled"} at collector startup — a restart/reconfigure artifact coincident with a "stopping metricbeat receiver" line, not persistent.
  • No actual panics in panic.log (misnamed — it's all info-level telemetry).

Questions for the community

  1. Is the component ID … contains '/', its EDOT diagnostics will be missing from the archive warning in 9.4.4's OTel manager a known issue for system/metrics inputs under the metricbeat. default: otel runtime? It looks like the only two components failing to register are the system/metrics ones, and they're the exact IDs flagged.
  2. Is the right fix to upgrade past 9.4.4 to a release that handles EDOT component IDs containing /, or is there a known workaround?
  3. As a workaround, is it supported/recommended to route just the system/metrics inputs back off the OTEL runtime (set metricbeat.default away from otel, or via policy runtime setting) so they run as plain metricbeat subprocesses? Any caveats for a Fleet-managed agent?
  4. The headline error is failed to connect to collector on the agent↔collector gRPC channel (localhost:6789, unix sockets under /opt/Elastic/Agent/data/tmp/*.sock). Anything specific to Ubuntu 24.04 (systemd sandboxing, AppArmor, socket perms, PrivateTmp) known to break this channel that I should check?

Any pointers would be hugely appreciated — happy to attach the full state.yaml / otel-merged.yaml / collector.log snippets if useful.

1 Like