Target specific pod in kubernetes filbeat

Elastic Stack Beats
1

I'm using this yaml to deploy filebeat to my GKE cluster:


it works like charm, but I want to push only container named "sy-prod"
so, I changed the configMap like below. 
filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            equals:
              kubernetes.container.name: sy-prod
          host: ${NODE_NAME}
          config:
            - type: container
              paths:
                - /var/log/containers/*${data.kubernetes.container.id}.log

but after I restart the filebeat, It can not find appropriate pods.
logs:

2020-05-26T12:27:10.097Z INFO [publisher] pipeline/module.go:97 Beat name: gke-sy-cluster-default-pool-db6b4068-fn8b2020-05-26T12:27:10.099Z INFO instance/beat.go:421 filebeat start running.2020-05-26T12:27:10.100Z INFO registrar/registrar.go:145 Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json2020-05-26T12:27:10.106Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s2020-05-26T12:27:10.109Z INFO add_cloud_metadata/add_cloud_metadata.go:351 add_cloud_metadata: hosting provider type detected as gcp, metadata={"availability_zone":"asia-northeast3-a","instance":{"id":"3493005511381915437","name":"gke-sy-cluster-default-pool-db6b4068-fn8b"},"machine":{"type":"n1-standard-1"},"project":{"id":"soyeon-275107"},"provider":"gcp"}2020-05-26T12:27:10.110Z INFO registrar/registrar.go:152 States Loaded from registrar: 162020-05-26T12:27:10.110Z INFO crawler/crawler.go:72 Loading Inputs: 02020-05-26T12:27:10.110Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 02020-05-26T12:27:10.110Z WARN [cfgwarn] kubernetes/kubernetes.go:55 BETA: The kubernetes autodiscover is beta2020-05-26T12:27:10.111Z INFO kubernetes/util.go:86 kubernetes: Using pod name gke-sy-cluster-default-pool-db6b4068-fn8b and namespace sy-elastic to discover kubernetes node2020-05-26T12:27:10.119Z ERROR kubernetes/util.go:90 kubernetes: Querying for pod failed with error: kubernetes api: Failure 404 pods "gke-sy-cluster-default-pool-db6b4068-fn8b" not found2020-05-26T12:27:10.119Z INFO [autodiscover] autodiscover/autodiscover.go:105 Starting autodiscover manager2020-05-26T12:27:10.119Z INFO kubernetes/watcher.go:182 kubernetes: Performing a resource sync for *v1.PodList2020-05-26T12:27:10.122Z INFO kubernetes/watcher.go:198 kubernetes: Resource sync done2020-05-26T12:27:10.122Z INFO kubernetes/watcher.go:242 kubernetes: Watching API for resource events2020-05-26T12:27:40.109Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":20,"time":{"ms":21}},"total":{"ticks":50,"time":{"ms":60},"value":50},"user":{"ticks":30,"time":{"ms":39}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":6},"info":{"ephemeral_id":"3b991c66-c7d1-4342-bbb1-6603b5d2baf0","uptime":{"ms":30054}},"memstats":{"gc_next":5787712,"memory_alloc":3146768,"memory_total":8570400,"rss":33861632},"runtime":{"goroutines":17}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":1},"load":{"1":0.1,"15":0.14,"5":0.17,"norm":{"1":0.1,"15":0.14,"5":0.17}}}}}}2020-05-26T12:28:10.108Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":20,"time":{"ms":1}},"total":{"ticks":60,"time":{"ms":3},"value":60},"user":{"ticks":40,"time":{"ms":2}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":6},"info":{"ephemeral_id":"3b991c66-c7d1-4342-bbb1-6603b5d2baf0","uptime":{"ms":60054}},"memstats":{"gc_next":5787712,"memory_alloc":3466408,"memory_total":8890040},"runtime":{"goroutines":17}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.06,"15":0.14,"5":0.16,"norm":{"1":0.06,"15":0.14,"5":0.16}}}}}}

sounds like filebeat confuses node name with pod names...
what am I doing wrong?

2

I tried that out and at least for my configuration I needed to change the namespace to my one kube-logging and interestingly your daemonset configuration didn't work for me.
I needed to change the API from
extensions/v1beta1
to
apps/v1

After that I still needed to add the hostname between type kubernetes and templates like that:

        - type: kubernetes
          host: ${HOSTNAME}
          templates:

And then it began to work for me.

3

OH MY GOD!!!!
I almost gave up on this!
I deleted everything and start with your solution...
and KABOOM! I could collect the logs what I wanted!

how did you figure out that host: ${HOSTNAME} is necessary??
You made my day! Thank you!

4

That's nice to hear. I just came over some configuration having the nodename there and short time ago had a similar issue and just figured out by accident.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.