We run 10 AutoOps OTel collectors (docker.elastic.co/elastic-agent/elastic-otel-collector-wolfi:9.5.5, the recommended autoops_es_ssl.yml config, metricbeatreceiver with autoops_es: cat_shards/node_stats/tasks_management @ 10s, cluster_health/cluster_settings @ 1m) on a 16 GiB / 2 vCPU Debian 13 VM, one collector per ES cluster.
Symptom: each collector's cgroup memory.current grows monotonically — ~500 MB at day 3 → 1.1–1.8 GB at day 24 per container — until the VM starves its page cache (systemd-journald: Under memory pressure, flushing caches, 126 msgs/day) and becomes unresponsive. No OOM kill ever fires (no limits, no swap).
Measured with read-only cgroup sampling every 10 min over multiple windows (full tables + CSVs available):
| collector | 12 h overnight: start → end | slope | R² |
|---|---|---|---|
| ut2-logs (tiny cluster) | 575 → 688 MB | +234 MB/day | 0.89 |
| fra-logs | 580 → 678 MB | +196 MB/day | 0.68 |
| eu-mpsearch | 557 → 628 MB | +130 MB/day | 0.91 |
| fleet total (10) | 5905 → 6523 MB | +1037 MB/day | — |
| host MemAvailable | 9771 → 9137 MB | −1035 MB/day | 0.94 |
Two things we'd like confirmed:
- Growth rate does not correlate with cluster size — our smallest cluster's collector grows fastest, the largest ones slowest/noisiest. Looks like per-scrape state accumulation rather than response volume.
GOMEMLIMIT=650MiB+mem_limit: 900Mcontains it (all ten plateau at 730–790 MB RSS, zero OOM/restarts in 8.4 days), but below the soft limit the heap still climbs ~100 MB/day. Is that expected GOGC behavior the collectors are designed around, or a leak inmetricbeatreceiver/autoops_es? Is GOMEMLIMIT the officially recommended mitigation?
- Version: elastic-otel-collector-wolfi 9.5.5 (reproduced on pre-9.5.5 too)
- OS: Debian 13, kernel 6.12.111, Docker 29.4.1, cgroup v2
Happy to provide raw CSVs and the config (secrets stripped).