Auditbeat, filebeat and journalbeat in kubernetes

Dmitriy_Esin · December 14, 2021, 1:11pm

Hi all!
I want to get an assistant with filebeat and auditbeat deployed in the k8s cluster.
Here are my yamls:

Auditbeat:

DmitriyEsin/elk_yml/blob/63b8d163e8e87c384604cfea4dc11c51adb6d7d4/auditbeat.yml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: auditbeat-config
  labels:
    k8s-app: auditbeat
data:
  auditbeat.yml: |-
    auditbeat.config.modules:
      # Mounted `auditbeat-daemonset-modules` configmap:
      path: ${path.config}/modules.d/*.yml
      # Reload module configs as they change:
      reload.enabled: false
    # When using containerd as runtime, a configuration like the following one
    # can be used to monitor files in containers using autodiscover.
    #auditbeat.autodiscover:
    #  providers:
    #  - type: kubernetes
    #    host: ${NODE_NAME}

This file has been truncated. show original

Filebeat:

github.com

DmitriyEsin/elk_yml/blob/63b8d163e8e87c384604cfea4dc11c51adb6d7d4/filebeat.yml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

This file has been truncated. show original

It runs successfully and has logs:

Auditbeat:

github.com

DmitriyEsin/elk_yml/blob/b2cb6f0cf26211eff3993509af18af3678dcdc44/auditbeat.log

2021-12-14T12:40:08.626Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":40375700}},"memory":{"mem":{"usage":{"bytes":-20480}}}},"cpu":{"system":{"ticks":35620,"time":{"ms":9}},"total":{"ticks":164320,"time":{"ms":39},"value":164320},"user":{"ticks":128700,"time":{"ms":30}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":14},"info":{"ephemeral_id":"83a9d7b5-28b0-43f8-86af-d34eb7f99de9","uptime":{"ms":72540097},"version":"7.14.0"},"memstats":{"gc_next":14795552,"memory_alloc":7825272,"memory_total":2835330464,"rss":116129792},"runtime":{"goroutines":57}},"libbeat":{"config":{"module":{"running":2}},"output":{"events":{"active":0}},"pipeline":{"clients":2,"events":{"active":0}}},"system":{"load":{"1":4.49,"15":4.61,"5":4.51,"norm":{"1":2.245,"15":2.305,"5":2.255}}}}}}
2021-12-14T12:40:38.626Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":174325057}},"memory":{"mem":{"usage":{"bytes":-8192}}}},"cpu":{"system":{"ticks":35640,"time":{"ms":21}},"total":{"ticks":164490,"time":{"ms":167},"value":164490},"user":{"ticks":128850,"time":{"ms":146}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":14},"info":{"ephemeral_id":"83a9d7b5-28b0-43f8-86af-d34eb7f99de9","uptime":{"ms":72570100},"version":"7.14.0"},"memstats":{"gc_next":14795552,"memory_alloc":8577552,"memory_total":2836082744,"rss":116121600},"runtime":{"goroutines":57}},"libbeat":{"config":{"module":{"running":2}},"output":{"events":{"active":0}},"pipeline":{"clients":2,"events":{"active":0}}},"system":{"load":{"1":4.67,"15":4.63,"5":4.57,"norm":{"1":2.335,"15":2.315,"5":2.285}}}}}}
2021-12-14T12:41:08.631Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":29128915}},"memory":{"mem":{"usage":{"bytes":131072}}}},"cpu":{"system":{"ticks":35660,"time":{"ms":13}},"total":{"ticks":164530,"time":{"ms":27},"value":164530},"user":{"ticks":128870,"time":{"ms":14}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":14},"info":{"ephemeral_id":"83a9d7b5-28b0-43f8-86af-d34eb7f99de9","uptime":{"ms":72600096},"version":"7.14.0"},"memstats":{"gc_next":14795552,"memory_alloc":10149592,"memory_total":2837654784,"rss":116121600},"runtime":{"goroutines":57}},"libbeat":{"config":{"module":{"running":2}},"output":{"events":{"active":0}},"pipeline":{"clients":2,"events":{"active":0}}},"system":{"load":{"1":4.57,"15":4.62,"5":4.55,"norm":{"1":2.285,"15":2.31,"5":2.275}}}}}}
2021-12-14T12:41:38.626Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":20069258}},"memory":{"mem":{"usage":{"bytes":-114688}}}},"cpu":{"system":{"ticks":35670,"time":{"ms":10}},"total":{"ticks":164540,"time":{"ms":20},"value":164540},"user":{"ticks":128870,"time":{"ms":10}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":14},"info":{"ephemeral_id":"83a9d7b5-28b0-43f8-86af-d34eb7f99de9","uptime":{"ms":72630098},"version":"7.14.0"},"memstats":{"gc_next":14795552,"memory_alloc":11646888,"memory_total":2839152080,"rss":116121600},"runtime":{"goroutines":57}},"libbeat":{"config":{"module":{"running":2}},"output":{"events":{"active":0}},"pipeline":{"clients":2,"events":{"active":0}}},"system":{"load":{"1":4.25,"15":4.59,"5":4.48,"norm":{"1":2.125,"15":2.295,"5":2.24}}}}}}
2021-12-14T12:42:08.626Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":47049215}}},"cpu":{"system":{"ticks":35680,"time":{"ms":9}},"total":{"ticks":164590,"time":{"ms":46},"value":164590},"user":{"ticks":128910,"time":{"ms":37}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":14},"info":{"ephemeral_id":"83a9d7b5-28b0-43f8-86af-d34eb7f99de9","uptime":{"ms":72660093},"version":"7.14.0"},"memstats":{"gc_next":14902496,"memory_alloc":7563344,"memory_total":2839605840,"rss":116121600},"runtime":{"goroutines":57}},"libbeat":{"config":{"module":{"running":2}},"output":{"events":{"active":0}},"pipeline":{"clients":2,"events":{"active":0}}},"system":{"load":{"1":4.28,"15":4.59,"5":4.47,"norm":{"1":2.14,"15":2.295,"5":2.235}}}}}}
2021-12-14T12:42:38.625Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":173212946}}},"cpu":{"system":{"ticks":35700,"time":{"ms":24}},"total":{"ticks":164770,"time":{"ms":179},"value":164770},"user":{"ticks":129070,"time":{"ms":155}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":14},"info":{"ephemeral_id":"83a9d7b5-28b0-43f8-86af-d34eb7f99de9","uptime":{"ms":72690096},"version":"7.14.0"},"memstats":{"gc_next":14902496,"memory_alloc":9164104,"memory_total":2841206600,"rss":116187136},"runtime":{"goroutines":57}},"libbeat":{"config":{"module":{"run

This file has been truncated. show original

Fiebeat:

github.com

DmitriyEsin/elk_yml/blob/e05e484ce9470916f93d39c29acd534e5184acc2/filebeat.log

2021-12-14T12:43:36.380Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":56445461}},"memory":{"mem":{"usage":{"bytes":65536}}}},"cpu":{"system":{"ticks":54850,"time":{"ms":22}},"total":{"ticks":152050,"time":{"ms":56},"value":152050},"user":{"ticks":97200,"time":{"ms":34}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":16},"info":{"ephemeral_id":"4890d608-3a56-42a2-855d-ed921b6a1aca","uptime":{"ms":72833462},"version":"7.14.0"},"memstats":{"gc_next":24593552,"memory_alloc":18830648,"memory_total":10258976488,"rss":126148608},"runtime":{"goroutines":81}},"filebeat":{"events":{"active":4,"added":13,"done":9},"harvester":{"open_files":5,"running":5}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":9,"active":0,"batches":5,"total":9},"read":{"bytes":2860},"write":{"bytes":35737}},"pipeline":{"clients":1,"events":{"active":4,"published":13,"total":13},"queue":{"acked":9}}},"registrar":{"states":{"current":32,"update":9},"writes":{"success":5,"total":5}},"system":{"load":{"1":4.87,"15":4.65,"5":4.66,"norm":{"1":2.435,"15":2.325,"5":2.33}}}}}}
2021-12-14T12:44:06.391Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":52472009}},"memory":{"mem":{"usage":{"bytes":94208}}}},"cpu":{"system":{"ticks":54870,"time":{"ms":27}},"total":{"ticks":152120,"time":{"ms":76},"value":152120},"user":{"ticks":97250,"time":{"ms":49}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":16},"info":{"ephemeral_id":"4890d608-3a56-42a2-855d-ed921b6a1aca","uptime":{"ms":72863473},"version":"7.14.0"},"memstats":{"gc_next":25030432,"memory_alloc":23625728,"memory_total":10263771568,"rss":126148608},"runtime":{"goroutines":81}},"filebeat":{"events":{"active":-4,"added":8,"done":12},"harvester":{"open_files":5,"running":5}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":12,"active":0,"batches":7,"total":12},"read":{"bytes":3864},"write":{"bytes":46996}},"pipeline":{"clients":1,"events":{"active":0,"published":8,"total":8},"queue":{"acked":12}}},"registrar":{"states":{"current":32,"update":12},"writes":{"success":7,"total":7}},"system":{"load":{"1":5.24,"15":4.68,"5":4.77,"norm":{"1":2.62,"15":2.34,"5":2.385}}}}}}
2021-12-14T12:44:36.383Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":74216580}},"memory":{"mem":{"usage":{"bytes":65536}}}},"cpu":{"system":{"ticks":54900,"time":{"ms":22}},"total":{"ticks":152180,"time":{"ms":48},"value":152180},"user":{"ticks":97280,"time":{"ms":26}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":16},"info":{"ephemeral_id":"4890d608-3a56-42a2-855d-ed921b6a1aca","uptime":{"ms":72893471},"version":"7.14.0"},"memstats":{"gc_next":24990272,"memory_alloc":15595848,"memory_total":10267199528,"rss":126148608},"runtime":{"goroutines":81}},"filebeat":{"events":{"added":11,"done":11},"harvester":{"open_files":5,"running":5}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":11,"active":0,"batches":5,"total":11},"read":{"bytes":3322},"write":{"bytes":42948}},"pipeline":{"clients":1,"events":{"active":0,"published":11,"total":11},"queue":{"acked":11}}},"registrar":{"states":{"current":32,"update":11},"writes":{"success":5,"total":5}},"system":{"load":{"1":5.03,"15":4.68,"5":4.76,"norm":{"1":2.515,"15":2.34,"5":2.38}}}}}}
2021-12-14T12:45:06.379Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":56416059}},"memory":{"mem":{"usage":{"bytes":94208}}}},"cpu":{"system":{"ticks":54910,"time":{"ms":15}},"total":{"ticks":152230,"time":{"ms":56},"value":152230},"user":{"ticks":97320,"time":{"ms":41}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":16},"info":{"ephemeral_id":"4890d608-3a56-42a2-855d-ed921b6a1aca","uptime":{"ms":72923463},"version":"7.14.0"},"memstats":{"gc_next":24990272,"memory_alloc":20248848,"memory_total":10271852528,"rss":126148608},"runtime":{"goroutines":81}},"filebeat":{"events":{"added":7,"done":7},"harvester":{"open_files":5,"running":5}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":7,"active":0,"batches":7,"total":7},"read":{"bytes":2709},"write":{"bytes":28974}},"pipeline":{"clients":1,"events":{"active":0,"published":7,"total":7},"queue":{"acked":7}}},"registrar":{"states":{"current":32,"update":7},"writes":{"success":7,"total":7}},"system":{"load":{"1":4.89,"15":4.68,"5":4.74,"norm":{"1":2.445,"15":2.34,"5":2.37}}}}}}
2021-12-14T12:45:36.389Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":41712272}},"memory":{"mem":{"usage":{"bytes":57344}}}},"cpu":{"system":{"ticks":54940,"time":{"ms":26}},"total":{"ticks":152290,"time":{"ms":61

This file has been truncated. show original

But I can't see any data from agents in kibana dashboards (kibana configured in yamls).

Also, I can't find any yaml for journalbeat on the official website:

Dmitriy_Esin · December 15, 2021, 6:24pm

I noticed that when I do ./elastic-agent inspect output -o default in the pod
Output is a default in spite of configured values in yaml.
Screenshot 2021-12-15 at 21.22.59
Any ideas?

system · January 12, 2022, 8:24pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.