Hey @exekias,
I have managed to almost achieve what I was after. I've been able to use the hints autodiscover to only publish logs with a specific annotation, whilst also being able to make use of multiline hints:
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
include_annotations: [logging']
processors:
- add_cloud_metadata: ~
- drop_event:
when:
not:
equals:
kubernetes.annotations.logging: 'true'
The last thing I'm struggling to get working is tail_files within the Kubernetes autodiscovery (Since our filebeat instances are stateless). Currently when a filebeat restarts it ends up scraping logs that have already been scraped, resulting in duplicated messages being published.
Is there somewhere I can configure the hints discovery/kubernetes logger to only tail files?
Neither of the below configs appear to work:
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
include_annotations: ['logging']
config:
- type: docker
tail_files: true
processors:
- add_cloud_metadata: ~
- drop_event:
when:
not:
equals:
kubernetes.annotations.logging: 'true'
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
include_annotations: ['logging']
tail_files: true
processors:
- add_cloud_metadata: ~
- drop_event:
when:
not:
equals:
kubernetes.annotations.logging: 'true'