# Autodiscover with hints, discover pods from all namespaces **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [September 10, 2018, 10:30am UTC](https://discuss.elastic.co/t/autodiscover-with-hints-discover-pods-from-all-namespaces/147939 "2018-09-10T10:30:45Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![havlan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/havlan/32/32379_2.png) [@havlan](https://discuss.elastic.co/u/havlan)\ **Post date:** [September 10, 2018, 10:30am UTC](https://discuss.elastic.co/t/autodiscover-with-hints-discover-pods-from-all-namespaces/147939/1 "2018-09-10T10:30:45Z")
Hi! I could not find any documentation regarding how to use autodiscover with hints and discover from all namespaces, is it possible to do this? Elastic stack: 6.4.0 Example Filebeat config: ```auto filebeat.autodiscover: providers: - type: kubernetes hints_enabled: true processors: - add_kubernetes_metadata: in_cluster: true # This ensures that every log that passes has required fields. #- drop_event.when.not: # has_fields: ['kubernetes.labels.ut-service', 'kubernetes.labels.ut-token', 'kubernetes.labels.ut-type'] - rename: fields: - from: "kubernetes.labels.ut-service" to: "_service" - from: "kubernetes.labels.ut-token" to: "_token" - from: "kubernetes.labels.ut-type" to: "type" output.logstash: hosts: ['${LOGSTASH_HOSTS}'] ``` Filebeat config only searches in kube-system namespace, the desirable functionality would be to specify which ones to take. ```auto 2018-09-10T09:40:55.618Z INFO kubernetes/util.go:86 kubernetes: Using pod name filebeat-9x7bk and namespace kube-system to discover kubernetes node ``` ---
**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\ **Post date:** [September 10, 2018, 2:15pm UTC](https://discuss.elastic.co/t/autodiscover-with-hints-discover-pods-from-all-namespaces/147939/2 "2018-09-10T14:15:49Z")
Hi @havian, Filebeat will collect logs from all namespaces, the message you are seeing is talking about the pod/namespace where filebeat is running, this is part of the initial discovery and configuration. Best regards ---
**Author:** ![havlan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/havlan/32/32379_2.png) [@havlan](https://discuss.elastic.co/u/havlan)\ **Post date:** [September 10, 2018, 2:31pm UTC](https://discuss.elastic.co/t/autodiscover-with-hints-discover-pods-from-all-namespaces/147939/3 "2018-09-10T14:31:47Z")
Thanks for resoponding so quickly. Would my example above work and use docker outputs for all new pods? Thanks. ---
**Author:** ![Alex\_Scoble](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_scoble/32/38142_2.png) [@Alex\_Scoble](https://discuss.elastic.co/u/Alex_Scoble)\ **Post date:** [September 11, 2018, 11:00pm UTC](https://discuss.elastic.co/t/autodiscover-with-hints-discover-pods-from-all-namespaces/147939/4 "2018-09-11T23:00:19Z")
I've found that you need to have it setup like so ``` filebeat.autodiscover: providers: - type: kubernetes hints.enabled: true include_annotations: '*' ``` in order to work...Your mileage may vary. --Alex ---
**Author:** ![havlan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/havlan/32/32379_2.png) [@havlan](https://discuss.elastic.co/u/havlan)\ **Post date:** [September 12, 2018, 6:11am UTC](https://discuss.elastic.co/t/autodiscover-with-hints-discover-pods-from-all-namespaces/147939/5 "2018-09-12T06:11:56Z")
@Alex_Scoble Thanks for your input. So to query against annotation one must include them, I guess that makes sense. I'm still having issues with actually getting logs to flow through, it seems like the autodiscover does not discover new events as they occur. I've experimented with different namespaces etc. But no success, my config is the one above. I also verified connection to logstash from minikube. (10.0.2.2:5044), so I'm pretty sure I just lack a working configuration. ```auto filebeat.yml: |- filebeat.autodiscover: providers: - type: kubernetes hints_enabled: true include_annotations: '*' processors: - add_kubernetes_metadata: in_cluster: true - drop_fields: fields: ["host"] output.logstash: hosts: ['${LOGSTASH_HOSTS}'] ``` Any input would be appreciated. -- Håvard ---
**Author:** ![Alex\_Scoble](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_scoble/32/38142_2.png) [@Alex\_Scoble](https://discuss.elastic.co/u/Alex_Scoble)\ **Post date:** [September 12, 2018, 4:26pm UTC](https://discuss.elastic.co/t/autodiscover-with-hints-discover-pods-from-all-namespaces/147939/6 "2018-09-12T16:26:20Z")
When you look at filebeat logs for any of the daemonset pods in kubernetes, what do they show? ---
**Author:** ![havlan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/havlan/32/32379_2.png) [@havlan](https://discuss.elastic.co/u/havlan)\ **Post date:** [September 18, 2018, 7:30am UTC](https://discuss.elastic.co/t/autodiscover-with-hints-discover-pods-from-all-namespaces/147939/7 "2018-09-18T07:30:46Z")
(WIth the filebeat config above) They don't show anything. I'm experimenting with minikube on the one end, and I have a docker ELK running on my laptop. So i'm sending filebeat logs to 10.0.2.2:5044, (I'm able to telnet at it and get an error output from logstash, so I have the necessary ports open). [Link to logs](https://gist.github.com/havlan/7d5795fb5ea3064766bd66cf004ea9b5) These logs are after restarting filebeat and restarting multiple applications, it seems like the harvester not fetching them at all. I'll be trying on a test cluster later to debug even further. Of course I might have missed the point about autodiscover with hints, but the kubernetes autodiscoverer worked so well with templates configured. ---
**Author:** ![havlan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/havlan/32/32379_2.png) [@havlan](https://discuss.elastic.co/u/havlan)\ **Post date:** [September 21, 2018, 6:42am UTC](https://discuss.elastic.co/t/autodiscover-with-hints-discover-pods-from-all-namespaces/147939/8 "2018-09-21T06:42:16Z")
@exekias @Alex_Scoble do you have any input for me? Any help would be amazing. ---
**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\ **Post date:** [September 25, 2018, 7:10am UTC](https://discuss.elastic.co/t/autodiscover-with-hints-discover-pods-from-all-namespaces/147939/9 "2018-09-25T07:10:34Z")
Hi @havlan, there are a few mistakes in your configuration: `hints_enabled: true` should be `hints.enabled: true` `include_annotations` should be a list although it's not required for your configuration to work. Best regards ---
**Author:** ![havlan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/havlan/32/32379_2.png) [@havlan](https://discuss.elastic.co/u/havlan)\ **Post date:** [September 25, 2018, 7:59am UTC](https://discuss.elastic.co/t/autodiscover-with-hints-discover-pods-from-all-namespaces/147939/10 "2018-09-25T07:59:25Z")
Thanks alot for pointing them out. About debugging the configuration; do you have any tips to doing that? I tried using debug flags, but I found it difficult to find errors in the logs. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 23, 2018, 7:59am UTC](https://discuss.elastic.co/t/autodiscover-with-hints-discover-pods-from-all-namespaces/147939/11 "2018-10-23T07:59:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.