Autodiscover with hints, discover pods from all namespaces

Elastic Stack Beats
1

Hi!
I could not find any documentation regarding how to use autodiscover with hints and discover from all namespaces, is it possible to do this?

Elastic stack: 6.4.0
Example Filebeat config:

filebeat.autodiscover:
      providers:
        - type: kubernetes
          hints_enabled: true

    processors:
      - add_kubernetes_metadata:
          in_cluster: true

      # This ensures that every log that passes has required fields.
      #- drop_event.when.not:
      #    has_fields: ['kubernetes.labels.ut-service', 'kubernetes.labels.ut-token', 'kubernetes.labels.ut-type']
      - rename:
          fields:
            - from: "kubernetes.labels.ut-service"
              to: "_service"
            - from: "kubernetes.labels.ut-token"
              to: "_token"
            - from: "kubernetes.labels.ut-type"
              to: "type"
    output.logstash:
      hosts: ['${LOGSTASH_HOSTS}']

Filebeat config only searches in kube-system namespace, the desirable functionality would be to specify which ones to take.

2018-09-10T09:40:55.618Z  INFO kubernetes/util.go:86 kubernetes: Using pod name filebeat-9x7bk and namespace kube-system to discover kubernetes node
2

Hi @havian,

Filebeat will collect logs from all namespaces, the message you are seeing is talking about the pod/namespace where filebeat is running, this is part of the initial discovery and configuration.

Best regards

3

Thanks for resoponding so quickly.
Would my example above work and use docker outputs for all new pods?

Thanks.

4

I've found that you need to have it setup like so

filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
      include_annotations: '*'

in order to work...Your mileage may vary.

--Alex

5

@Alex_Scoble
Thanks for your input.
So to query against annotation one must include them, I guess that makes sense.

I'm still having issues with actually getting logs to flow through, it seems like the autodiscover does not discover new events as they occur. I've experimented with different namespaces etc. But no success, my config is the one above. I also verified connection to logstash from minikube. (10.0.2.2:5044), so I'm pretty sure I just lack a working configuration.

filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          hints_enabled: true
          include_annotations: '*'

    processors:
      - add_kubernetes_metadata:
          in_cluster: true
      - drop_fields:
          fields: ["host"]
    output.logstash:
      hosts: ['${LOGSTASH_HOSTS}']

Any input would be appreciated.

-- Håvard

6

When you look at filebeat logs for any of the daemonset pods in kubernetes, what do they show?

7

(WIth the filebeat config above) They don't show anything. I'm experimenting with minikube on the one end, and I have a docker ELK running on my laptop.
So i'm sending filebeat logs to 10.0.2.2:5044, (I'm able to telnet at it and get an error output from logstash, so I have the necessary ports open).
Link to logs
These logs are after restarting filebeat and restarting multiple applications, it seems like the harvester not fetching them at all.
I'll be trying on a test cluster later to debug even further.
Of course I might have missed the point about autodiscover with hints, but the kubernetes autodiscoverer worked so well with templates configured.

8

@exekias @Alex_Scoble do you have any input for me? Any help would be amazing.

9

Hi @havlan, there are a few mistakes in your configuration:

hints_enabled: true should be hints.enabled: true

include_annotations should be a list although it's not required for your configuration to work.

Best regards

10

Thanks alot for pointing them out.
About debugging the configuration; do you have any tips to doing that? I tried using debug flags, but I found it difficult to find errors in the logs.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.