We are running Filebeat in Kubernetes in autodiscover mode. When configuring the filestream input type with a new registry data path (so clean state from the beginning) and setting ignore_inactive to since_last_start it still reads files from the beginning.
The doc though states the following (filestream input | Filebeat Reference [8.15] | Elastic):
For files that were never seen before, the offset state is set to the end of the file.
Tested configuration:
filebeat.registry.path: ${path.data}/logs-registry
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
hints.default_config:
type: filestream
id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
paths:
- /var/log/pods/${data.kubernetes.namespace}_${data.kubernetes.pod.name}_${data.kubernetes.pod.uid}/${data.kubernetes.container.name}/*.log
ignore_inactive: since_last_start
parsers:
- container: ~
prospector:
scanner:
fingerprint.enabled: true
symlinks: false
file_identity.fingerprint: ~