# Filebeat Autodiscover Hints Breaking Template **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [June 25, 2018, 6:58pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-hints-breaking-template/137310 "2018-06-25T18:58:07Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Evesy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evesy/32/29520_2.png) [@Evesy](https://discuss.elastic.co/u/Evesy)\ **Post date:** [June 25, 2018, 6:58pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-hints-breaking-template/137310/1 "2018-06-25T18:58:07Z")
Hi, We're using the below to scrape Kubernetes logs based on the presence of a specific annotation: ```auto filebeat.autodiscover: providers: - type: kubernetes in_cluster: true tags: - "kubernetes" templates: - condition: contains: kubernetes.annotations.logging: "true" config: - type: docker tail_files: true containers.ids: - "${data.kubernetes.container.id}" processors: - add_kubernetes_metadata: in_cluster: true - add_cloud_metadata: ~ ``` This on its own is working as expected, we get logs for pods that have the `logging` annotation on them, with cloud metadata attached to the event. We thought we could add `hints.enabled: true` to the configuration that would then allow certain pods to specify custom parsing operations (i.e. to handle the presence of multiline logs). However when adding `hints.enabled: true` to the provider configuration (example below), suddenly _everything_ from Kubernetes starts getting logged, and is also missing the cloud metadata: ```auto - type: kubernetes hints.enabled: true in_cluster: true ``` What we're after is the ability to send logs _only_ from pods that have a specific annotation, and then provide a further option on top of that (using hints) to provide annotations to help better parse multiline messages etc. Is that possible? Cheers, Mike ---
**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\ **Post date:** [June 25, 2018, 9:17pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-hints-breaking-template/137310/2 "2018-06-25T21:17:21Z")
Hi @Evesy, When you are using hints, all logs are retrieved by default, as you said. But you can change that behavior, as templates are evaluated first, for instance creating a template that throws no configs. Also, there is a hint to disable logging, try: `co.elastic.logs/disable: true`. I'll have a look to the docs, we may need to include this one 😇 Best regards ---
**Author:** ![Evesy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evesy/32/29520_2.png) [@Evesy](https://discuss.elastic.co/u/Evesy)\ **Post date:** [June 25, 2018, 9:21pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-hints-breaking-template/137310/3 "2018-06-25T21:21:14Z")
Hey @exekias Thanks very much for the quick response! I'm pretty new to using Filebeat, would you possible be able to clarify what you mean by this? > for instance creating a template that throws no configs Are there any plans/would a feature request be welcomed to have an inverse of the current behaviour, with an option to default no log collection by hints, and only enabled when `co.elastic.logs/enable: true` is set? ---
**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\ **Post date:** [June 25, 2018, 9:23pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-hints-breaking-template/137310/4 "2018-06-25T21:23:50Z")
Sorry, I was not clear enough :), I was thinking on a config like this: ```auto filebeat.autodiscover: providers: - type: kubernetes in_cluster: true hints.enabled: true templates: - condition: contains: kubernetes.annotations.logging: "false" config: ``` As templates are processed first, if `logging == false` hints won't be processed. In practice this removes the need for adding it as a feature. As you can use these settings to get the behavior you want. BTW: I just opened a PR for the docs: [https://github.com/elastic/beats/pull/7406](https://github.com/elastic/beats/pull/7406) ---
**Author:** ![Evesy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evesy/32/29520_2.png) [@Evesy](https://discuss.elastic.co/u/Evesy)\ **Post date:** [June 25, 2018, 9:32pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-hints-breaking-template/137310/5 "2018-06-25T21:32:06Z")
@exekias Is this block in the example supposed to have a not statement somewhere: ```auto - condition: contains: kubernetes.annotations.logging: "false" ``` As I understand it from that example, files will be processed for anything with `logging == false` Apologies if I'm missing the point (quite likely), ideally I don't want to rely on everything having to include an annotation to be _excluded_ from logging, I'd rather that be the default and there be an annotation to opt in Thanks again Mike ---
**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\ **Post date:** [June 25, 2018, 9:36pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-hints-breaking-template/137310/6 "2018-06-25T21:36:38Z")
ok then, something like this would do (I believe, to be tested): ```auto filebeat.autodiscover: providers: - type: kubernetes in_cluster: true hints.enabled: true templates: - condition: not.contains: kubernetes.annotations.logging: "true" config: ``` ---
**Author:** ![Evesy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evesy/32/29520_2.png) [@Evesy](https://discuss.elastic.co/u/Evesy)\ **Post date:** [June 25, 2018, 9:48pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-hints-breaking-template/137310/7 "2018-06-25T21:48:58Z")
@exekias As soon as I add `hints.enabled: true` I start seeing all files being processed. I've tried with both the below templates: ```auto templates: - condition: contains: kubernetes.annotations.logging: "true" config: ``` (Only process logs with `logging == true`) ```auto templates: - condition: not.contains: kubernetes.annotations.logging: "true" config: ``` (Only process logs where `logging != true`) ---
**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\ **Post date:** [June 25, 2018, 10:10pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-hints-breaking-template/137310/8 "2018-06-25T22:10:02Z")
Uhm, I think you are right, after checking the code, hints will work as long as you don't provide a valid config. This is useful to override some behavior, but not to cancel it entirely. In this case, I would be OK with accepting a feature or pull request to support what you need. Feel free to open a new issue, please give as much detail as possible: [https://github.com/elastic/beats/issues](https://github.com/elastic/beats/issues) Best regards ---
**Author:** ![Evesy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evesy/32/29520_2.png) [@Evesy](https://discuss.elastic.co/u/Evesy)\ **Post date:** [June 25, 2018, 10:24pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-hints-breaking-template/137310/9 "2018-06-25T22:24:50Z")
Thank you for the help @exekias, it's much appreciated. I have raised [https://github.com/elastic/beats/issues/7407](https://github.com/elastic/beats/issues/7407) in relation to this discussion Thanks, Mike ---
**Author:** ![Evesy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evesy/32/29520_2.png) [@Evesy](https://discuss.elastic.co/u/Evesy)\ **Post date:** [June 26, 2018, 3:56pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-hints-breaking-template/137310/10 "2018-06-26T15:56:07Z")
Hey @exekias, I have managed to almost achieve what I was after. I've been able to use the hints autodiscover to only publish logs with a specific annotation, whilst also being able to make use of multiline hints: ```auto filebeat.autodiscover: providers: - type: kubernetes hints.enabled: true include_annotations: [logging'] processors: - add_cloud_metadata: ~ - drop_event: when: not: equals: kubernetes.annotations.logging: 'true' ``` The last thing I'm struggling to get working is `tail_files` within the Kubernetes autodiscovery (Since our filebeat instances are stateless). Currently when a filebeat restarts it ends up scraping logs that have already been scraped, resulting in duplicated messages being published. Is there somewhere I can configure the hints discovery/kubernetes logger to only tail files? Neither of the below configs appear to work: ```auto filebeat.autodiscover: providers: - type: kubernetes hints.enabled: true include_annotations: ['logging'] config: - type: docker tail_files: true processors: - add_cloud_metadata: ~ - drop_event: when: not: equals: kubernetes.annotations.logging: 'true' ``` ```auto filebeat.autodiscover: providers: - type: kubernetes hints.enabled: true include_annotations: ['logging'] tail_files: true processors: - add_cloud_metadata: ~ - drop_event: when: not: equals: kubernetes.annotations.logging: 'true' ``` ---
**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\ **Post date:** [June 26, 2018, 3:58pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-hints-breaking-template/137310/11 "2018-06-26T15:58:08Z")
Nice, I'm glad that did the trick. You don't need to use `tail_files`. Filebeat can keep a registry of what's sent and what's pending, you just need to update this volume to use a hostPath: [https://github.com/elastic/beats/blob/master/deploy/kubernetes/filebeat-kubernetes.yaml#L117-L121](https://github.com/elastic/beats/blob/master/deploy/kubernetes/filebeat-kubernetes.yaml#L117-L121) Best regards ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 24, 2018, 3:58pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-hints-breaking-template/137310/12 "2018-07-24T15:58:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.