Filebeat occasionally loses logs: Dropping event: no topic could be selected (Kubernetes + Kafka)

Elastic Stack Beats
1

Hi everyone,
I’ve been troubleshooting an issue for the last few days and would like to confirm whether this behavior is expected or if there is a workaround.

Problem

When Filebeat starts on a Kubernetes node (DaemonSet), sometimes it drops container logs produced in the first few seconds of a pod’s lifecycle. The logs never reach Kafka and cannot be recovered.

The only message in the Filebeat logs during the loss window is:

2025-11-04T11:48:19.764Z	ERROR	[kafka]	kafka/client.go:147	Dropping event: no topic could be selected
2025-11-04T11:48:19.764Z	ERROR	[kafka]	kafka/client.go:147	Dropping event: no topic could be selected
2025-11-04T11:48:19.764Z	ERROR	[kafka]	kafka/client.go:147	Dropping event: no topic could be selected
2025-11-04T11:48:19.764Z	ERROR	[kafka]	kafka/client.go:147	Dropping event: no topic could be selected
2025-11-04T11:48:19.764Z	ERROR	[kafka]	kafka/client.go:147	Dropping event: no topic could be selected

After a few seconds, Filebeat resumes sending logs normally. Only the early logs are gone.

Environment

Component Value
Filebeat version 7.10.2
Platform Kubernetes
Deployment DaemonSet
Log source /var/log/containers/*.log
Output Kafka

Current Config (simplified)

filebeat.inputs:
- type: container
  enabled: true
  paths:
    - '/var/log/containers/*.log'
  processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: "/var/log/containers/"
output.kafka:
  hosts: ["..."]
  username: "..."
  password: "..."
  topic: "%{[kubernetes.namespace]}"

This looks like a race condition: Filebeat starts harvesting logs before Kubernetes metadata is resolved, causing topic interpolation to fail → message dropped.

Questions

  1. Is this an expected behavior today?

  2. Any recommended config pattern to guarantee zero log loss when using metadata-based topic routing? not like topic: "filebeat-%{[kubernetes.namespace]:pending}"

Happy to provide more info if helpful

I can test patches, try alternative configs, or collect extra debug logs if needed.

Thanks a lot!