Hi.
I have one Pod with running Heartbeat in our K8s cluster and tried to setup autodiscovery, but without any success...
kluvi@kluvi:~$ k log -f deployment/service-monitoring-heartbeat-autodiscover
log is DEPRECATED and will be removed in a future version. Use logs instead.
2020-01-02T21:14:39.139Z INFO instance/beat.go:610 Home path: [/usr/share/heartbeat] Config path: [/usr/share/heartbeat] Data path: [/usr/share/heartbeat/data] Logs path: [/usr/share/heartbeat/logs]
2020-01-02T21:14:39.139Z DEBUG [beat] instance/beat.go:662 Beat metadata path: /usr/share/heartbeat/data/meta.json
2020-01-02T21:14:39.140Z INFO instance/beat.go:618 Beat ID: f75d791a-4033-4e36-8580-33604a49b4b8
2020-01-02T21:14:39.149Z INFO add_kubernetes_metadata/kubernetes.go:68 add_kubernetes_metadata: kubernetes env detected, with version: v1.15.3
2020-01-02T21:14:39.149Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:130 add_kubernetes_metadata: could not initialize kubernetes plugin with zero matcher plugins
2020-01-02T21:14:39.149Z DEBUG [processors] processors/processor.go:101 Generated new processors: add_kubernetes_metadata
2020-01-02T21:14:39.150Z INFO [api] api/server.go:62 Starting stats endpoint
2020-01-02T21:14:39.151Z INFO [api] api/server.go:64 Metrics endpoint listening on: 127.0.0.1:5066 (configured: localhost)
2020-01-02T21:14:39.150Z DEBUG [seccomp] seccomp/seccomp.go:117 Loading syscall filter {"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchown","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}}
2020-01-02T21:14:39.151Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2020-01-02T21:14:39.151Z INFO [beat] instance/beat.go:941 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/heartbeat", "data": "/usr/share/heartbeat/data", "home": "/usr/share/heartbeat", "logs": "/usr/share/heartbeat/logs"}, "type": "heartbeat", "uuid": "f75d791a-4033-4e36-8580-33604a49b4b8"}}}
2020-01-02T21:14:39.151Z INFO [beat] instance/beat.go:950 Build info {"system_info": {"build": {"commit": "6d0d0ae079e5cb1d4f224801ac6df926dfb1594c", "libbeat": "7.5.0", "time": "2019-11-25T23:40:34.000Z", "version": "7.5.0"}}}
2020-01-02T21:14:39.151Z INFO [beat] instance/beat.go:953 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":64,"version":"go1.12.12"}}}
2020-01-02T21:14:39.170Z INFO [beat] instance/beat.go:957 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-12-23T15:15:23Z","containerized":false,"name":"service-monitoring-heartbeat-autodiscover-dd8f5889-rzjlw","ip":["127.0.0.1/8","10.233.91.39/32"],"kernel_version":"3.10.0-1062.9.1.el7.x86_64","mac":["82:49:80:6d:fb:c2"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":7,"patch":1908,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-01-02T21:14:39.170Z INFO [beat] instance/beat.go:986 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/heartbeat", "exe": "/usr/share/heartbeat/heartbeat", "name": "heartbeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-01-02T21:14:38.080Z"}}}
2020-01-02T21:14:39.170Z INFO instance/beat.go:297 Setup Beat: heartbeat; Version: 7.5.0
2020-01-02T21:14:39.170Z DEBUG [beat] instance/beat.go:323 Initializing output plugins
2020-01-02T21:14:39.171Z INFO [index-management] idxmgmt/std.go:182 Set output.elasticsearch.index to 'heartbeat-7.5.0' as ILM is enabled.
2020-01-02T21:14:39.171Z INFO elasticsearch/client.go:171 Elasticsearch url: http://service-monitoring-elasticsearch.bf.svc:9200
2020-01-02T21:14:39.171Z DEBUG [publisher] pipeline/consumer.go:137 start pipeline event consumer
2020-01-02T21:14:39.171Z INFO [publisher] pipeline/module.go:97 Beat name: autodiscover
2020-01-02T21:14:39.171Z INFO instance/beat.go:429 heartbeat start running.
2020-01-02T21:14:39.171Z INFO beater/heartbeat.go:80 heartbeat is running! Hit CTRL-C to stop it.
2020-01-02T21:14:39.171Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-01-02T21:14:39.171Z WARN [cfgwarn] kubernetes/kubernetes.go:59 BETA: The kubernetes autodiscover is beta
2020-01-02T21:14:39.172Z INFO kubernetes/util.go:79 kubernetes: Using node kubernetes.default.svc provided in the config
2020-01-02T21:14:39.172Z DEBUG [autodiscover] kubernetes/kubernetes.go:84 Initializing a new Kubernetes watcher using host: kubernetes.default.svc
2020-01-02T21:14:39.172Z DEBUG [conditions] conditions/conditions.go:98 New condition contains: map[]
2020-01-02T21:14:39.173Z DEBUG [autodiscover] autodiscover/autodiscover.go:82 Configured autodiscover provider: kubernetes
2020-01-02T21:14:39.173Z INFO [autodiscover] autodiscover/autodiscover.go:104 Starting autodiscover manager
2020-01-02T21:14:39.273Z DEBUG [kubernetes] kubernetes/watcher.go:241 cache sync done
2020-01-02T21:14:39.273Z DEBUG [scheduler] scheduler/scheduler.go:186 Start scheduler.
2020-01-02T21:15:09.175Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":30,"time":{"ms":38}},"total":{"ticks":110,"time":{"ms":125},"value":110},"user":{"ticks":80,"time":{"ms":87}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"0c02389c-1770-40b0-a2f2-6f5a7734fe62","uptime":{"ms":30049}},"memstats":{"gc_next":9020496,"memory_alloc":4639840,"memory_total":9716216,"rss":38191104},"runtime":{"goroutines":26}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":64},"load":{"1":14.33,"15":16.71,"5":16.23,"norm":{"1":0.2239,"15":0.2611,"5":0.2536}}}}}}
Config (part of it - it does not contain any heartbeat.monitor):
heartbeat.autodiscover:
providers:
- type: kubernetes
host: kubernetes.default.svc
namespace: bf
resource: service
scope: cluster
include_annotations: ["cz.bdfx.heartbeat.type"]
templates:
- condition:
contains:
kubernetes.annotations.cz.bdfx.heartbeat.type: "http"
config:
- type: http
enabled: true
name: "${data.kubernetes.annotations.cz.bdfx.heartbeat.name}"
mode: any
hosts: ["${data.kubernetes.service.name}.bf.svc:${data.kubernetes.annotations.cz.bdfx.heartbeat.port}"]
schedule: "@every ${data.kubernetes.annotations.cz.bdfx.heartbeat.seconds}s"
timeout: 16s
max_redirects: 0
check:
request:
method: GET
response:
status: 200
Service:
kluvi@kluvi:~$ k get svc service-monitoring-kibana -o yaml
apiVersion: v1
kind: Service
metadata:
annotations:
cz.bdfx.heartbeat.name: Kibana @monitoring
cz.bdfx.heartbeat.port: "80"
cz.bdfx.heartbeat.seconds: "10"
cz.bdfx.heartbeat.type: http
...
Another Pod with another Heartbeat (but manually configured monitors) works well.