Metricbeat Autodiscover Redis Module No Data

Elastic Stack Beats
1

For some reason I cannot pick up redis data using autodiscover for metricbeat. Auto discover for nginx is working just fine. I am running out of troubleshooting ideas.

k8s version 1.14.6
metricbeats conf snip

metricbeat.autodiscover:
      providers:
        - type: kubernetes
          host: ${HOSTNAME}
          in_cluster: true
          templates:
            - condition:
                contains:
                  kubernetes.container.image: nginx
              config:
                - module: nginx
                  metricsets: ["stubstatus"]
                  enable: true
                  period: 10s
                  hosts: ["${data.host}:18080"]
                  server_status_path: "nginx_status"
            - condition:
                contains:
                  kubernetes.container.image: redis
              config:
                - module: redis
                  metricsets: ["info", "keyspace"]
                  period: 10s
                  hosts: ["${data.host}:6379"]

Some strange log entries from metricbeat, may not be related...

E0110 21:09:57.785127       1 reflector.go:342] github.com/elastic/beats/libbeat/common/kubernetes/watcher.go:235: expected type *v1beta1.ReplicaSet, but watch event object had type *v1.ReplicaSet
2

Can you run Metricbeat with logging.level: debug and share your Metricbeat logs right after starting it up (say, the first 30 seconds or so)? Please make sure to scrub any sensitive information in the logs before posting them here.

1 Like
3

@shaunak

Here you go...

❯ k logs metricbeat-metricbeat-kbrlq
2020-01-13T18:15:01.615Z	INFO	instance/beat.go:610	Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]
2020-01-13T18:15:01.616Z	INFO	instance/beat.go:618	Beat ID: ffae0832-5077-40ee-aa54-a09455c475da
2020-01-13T18:15:01.637Z	INFO	[api]	api/server.go:62	Starting stats endpoint
2020-01-13T18:15:01.637Z	INFO	[api]	api/server.go:64	Metrics endpoint listening on: 127.0.0.1:5066 (configured: localhost)
2020-01-13T18:15:01.638Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2020-01-13T18:15:01.638Z	INFO	[beat]	instance/beat.go:941	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat", "data": "/usr/share/metricbeat/data", "home": "/usr/share/metricbeat", "logs": "/usr/share/metricbeat/logs"}, "type": "metricbeat", "uuid": "ffae0832-5077-40ee-aa54-a09455c475da"}}}
2020-01-13T18:15:01.638Z	INFO	[beat]	instance/beat.go:950	Build info	{"system_info": {"build": {"commit": "6d0d0ae079e5cb1d4f224801ac6df926dfb1594c", "libbeat": "7.5.0", "time": "2019-11-26T00:09:41.000Z", "version": "7.5.0"}}}
2020-01-13T18:15:01.638Z	INFO	[beat]	instance/beat.go:953	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.12.12"}}}
2020-01-13T18:15:01.640Z	INFO	[beat]	instance/beat.go:957	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-01-08T06:57:13Z","containerized":false,"name":"metricbeat-metricbeat-kbrlq","ip":["127.0.0.1/8","172.16.0.166/22"],"kernel_version":"4.15.0-1066-azure","mac":["ce:22:d4:ff:50:b4"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":7,"patch":1908,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-01-13T18:15:01.641Z	INFO	[beat]	instance/beat.go:986	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/metricbeat", "exe": "/usr/share/metricbeat/metricbeat", "name": "metricbeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-01-13T18:15:00.520Z"}}}
2020-01-13T18:15:01.641Z	INFO	instance/beat.go:297	Setup Beat: metricbeat; Version: 7.5.0
2020-01-13T18:15:01.641Z	INFO	[index-management]	idxmgmt/std.go:182	Set output.elasticsearch.index to 'metricbeat-7.5.0' as ILM is enabled.
2020-01-13T18:15:01.642Z	INFO	elasticsearch/client.go:171	Elasticsearch url: http://elasticsearch-master:9200
2020-01-13T18:15:01.643Z	INFO	[publisher]	pipeline/module.go:97	Beat name: metricbeat-metricbeat-kbrlq
2020-01-13T18:15:01.748Z	INFO	add_kubernetes_metadata/kubernetes.go:68	add_kubernetes_metadata: kubernetes env detected, with version: v1.14.6
2020-01-13T18:15:01.748Z	INFO	kubernetes/util.go:94	kubernetes: Using pod name metricbeat-metricbeat-kbrlq and namespace elk to discover kubernetes node
2020-01-13T18:15:01.763Z	INFO	kubernetes/util.go:100	kubernetes: Using node aks-agentpool-11078474-5 discovered by in cluster pod node query
2020-01-13T18:15:01.870Z	INFO	kubernetes/util.go:79	kubernetes: Using node aks-agentpool-11078474-5 provided in the config
2020-01-13T18:15:01.876Z	INFO	add_cloud_metadata/add_cloud_metadata.go:93	add_cloud_metadata: hosting provider type detected as az, metadata={"instance":{"id":"ad49b400-a6f6-4041-896c-4e8315e254a2","name":"aks-agentpool-11078474-5"},"machine":{"type":"Standard_D2s_v3"},"provider":"az","region":"westus"}
2020-01-13T18:15:01.876Z	INFO	kubernetes/util.go:79	kubernetes: Using node aks-agentpool-11078474-5 provided in the config
2020-01-13T18:15:01.906Z	INFO	filesystem/filesystem.go:57	Ignoring filesystem types: sysfs, rootfs, ramfs, bdev, proc, cpuset, cgroup, cgroup2, tmpfs, devtmpfs, configfs, debugfs, tracefs, securityfs, sockfs, dax, bpf, pipefs, hugetlbfs, devpts, ecryptfs, fuse, fusectl, pstore, mqueue, autofs, rpc_pipefs, aufs, overlay, cifs
2020-01-13T18:15:01.933Z	INFO	[system.fsstat]	fsstat/fsstat.go:56	Ignoring filesystem types: %ssysfs, rootfs, ramfs, bdev, proc, cpuset, cgroup, cgroup2, tmpfs, devtmpfs, configfs, debugfs, tracefs, securityfs, sockfs, dax, bpf, pipefs, hugetlbfs, devpts, ecryptfs, fuse, fusectl, pstore, mqueue, autofs, rpc_pipefs, aufs, overlay, cifs
2020-01-13T18:15:01.933Z	WARN	[cfgwarn]	kubernetes/kubernetes.go:59	BETA: The kubernetes autodiscover is beta
2020-01-13T18:15:01.936Z	INFO	kubernetes/util.go:79	kubernetes: Using node metricbeat-metricbeat-kbrlq provided in the config
2020-01-13T18:15:01.936Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2020-01-13T18:15:01.936Z	INFO	kibana/client.go:117	Kibana url: http://10.0.252.214:5601
2020-01-13T18:15:02.640Z	INFO	kibana/client.go:117	Kibana url: http://10.0.252.214:5601
2020-01-13T18:15:31.938Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":80,"time":{"ms":85}},"total":{"ticks":990,"time":{"ms":1003},"value":990},"user":{"ticks":910,"time":{"ms":918}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":11},"info":{"ephemeral_id":"3d53621e-303d-4f58-8ac0-c4a4c0755ade","uptime":{"ms":30407}},"memstats":{"gc_next":20625040,"memory_alloc":16071176,"memory_total":99791200,"rss":95068160},"runtime":{"goroutines":54}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":2},"load":{"1":2.02,"15":2.34,"5":2.74,"norm":{"1":1.01,"15":1.17,"5":1.37}}}}}}
2020-01-13T18:15:57.249Z	INFO	instance/beat.go:780	Kibana dashboards successfully loaded.
2020-01-13T18:15:57.249Z	INFO	instance/beat.go:429	metricbeat start running.
2020-01-13T18:15:57.249Z	INFO	[autodiscover]	autodiscover/autodiscover.go:104	Starting autodiscover manager
2020-01-13T18:15:58.640Z	INFO	pipeline/output.go:95	Connecting to backoff(elasticsearch(http://elasticsearch-master:9200))
2020-01-13T18:15:58.645Z	INFO	elasticsearch/client.go:753	Attempting to connect to Elasticsearch version 7.5.0
2020-01-13T18:15:58.679Z	INFO	[index-management]	idxmgmt/std.go:256	Auto ILM enable success.
2020-01-13T18:15:58.681Z	INFO	[index-management.ilm]	ilm/std.go:138	do not generate ilm policy: exists=true, overwrite=false
2020-01-13T18:15:58.681Z	INFO	[index-management]	idxmgmt/std.go:269	ILM policy successfully loaded.
2020-01-13T18:15:58.681Z	INFO	[index-management]	idxmgmt/std.go:408	Set setup.template.name to '{metricbeat-7.5.0 {now/d}-000001}' as ILM is enabled.
2020-01-13T18:15:58.682Z	INFO	[index-management]	idxmgmt/std.go:413	Set setup.template.pattern to 'metricbeat-7.5.0-*' as ILM is enabled.
2020-01-13T18:15:58.682Z	INFO	[index-management]	idxmgmt/std.go:447	Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.5.0 {now/d}-000001} as ILM is enabled.
2020-01-13T18:15:58.682Z	INFO	[index-management]	idxmgmt/std.go:451	Set settings.index.lifecycle.name in template to {metricbeat-7.5.0 {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-01-13T18:15:58.688Z	INFO	template/load.go:89	Template metricbeat-7.5.0 already exists and will not be overwritten.
2020-01-13T18:15:58.688Z	INFO	[index-management]	idxmgmt/std.go:293	Loaded index template.
2020-01-13T18:15:58.696Z	INFO	[index-management]	idxmgmt/std.go:304	Write alias successfully generated.
2020-01-13T18:15:58.696Z	INFO	pipeline/output.go:105	Connection to backoff(elasticsearch(http://elasticsearch-master:9200)) established
2020-01-13T18:16:01.938Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":170,"time":{"ms":90}},"total":{"ticks":1970,"time":{"ms":976},"value":1970},"user":{"ticks":1800,"time":{"ms":886}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":17},"info":{"ephemeral_id":"3d53621e-303d-4f58-8ac0-c4a4c0755ade","uptime":{"ms":60407}},"memstats":{"gc_next":30421392,"memory_alloc":15307272,"memory_total":196770096,"rss":4677632},"runtime":{"goroutines":157}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":146,"batches":5,"total":146},"read":{"bytes":5233},"write":{"bytes":145123}},"pipeline":{"clients":4,"events":{"active":232,"filtered":1,"published":378,"retry":50,"total":379},"queue":{"acked":146}}},"metricbeat":{"kubernetes":{"node":{"events":1,"success":1},"pod":{"events":18,"success":18},"state_container":{"events":194,"success":194},"state_replicaset":{"events":123,"success":123},"volume":{"events":38,"success":38}},"system":{"filesystem":{"events":1,"success":1},"load":{"events":1,"success":1},"network":{"events":2,"success":2},"process":{"events":1,"success":1}}},"system":{"load":{"1":1.42,"15":2.28,"5":2.52,"norm":{"1":0.71,"15":1.14,"5":1.26}}}}}}
2020-01-13T18:16:31.939Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":350,"time":{"ms":184}},"total":{"ticks":5070,"time":{"ms":3104},"value":5070},"user":{"ticks":4720,"time":{"ms":2920}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":22},"info":{"ephemeral_id":"3d53621e-303d-4f58-8ac0-c4a4c0755ade","uptime":{"ms":90408}},"memstats":{"gc_next":56599632,"memory_alloc":34073032,"memory_total":561006192,"rss":26832896},"runtime":{"goroutines":203}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":1862,"batches":49,"total":1862},"read":{"bytes":28544},"write":{"bytes":2099770}},"pipeline":{"clients":4,"events":{"active":0,"published":1630,"total":1630},"queue":{"acked":1862}}},"metricbeat":{"kubernetes":{"container":{"events":66,"success":66},"event":{"events":2,"success":2},"node":{"events":3,"success":3},"pod":{"events":54,"success":54},"state_container":{"events":388,"success":388},"state_deployment":{"events":153,"success":153},"state_node":{"events":18,"success":18},"state_pod":{"events":432,"success":432},"state_replicaset":{"events":369,"success":369},"system":{"events":9,"success":9},"volume":{"events":114,"success":114}},"system":{"cpu":{"events":3,"success":3},"fsstat":{"events":1,"success":1},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":6,"success":6},"process":{"events":3,"success":3},"process_summary":{"events":3,"success":3}}},"system":{"load":{"1":0.92,"15":2.21,"5":2.29,"norm":{"1":0.46,"15":1.105,"5":1.145}}}}}}
4

Hmm, I'm not seeing any DEBUG level messages in that log snipped. Are you sure you set logging.level: debug before starting up Metricbeat and gathering those logs?

5

hmmmm...I set it here, but this may not be the correct place.

output.elasticsearch:
      hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
    setup.kibana:
      host: '${KIBANA_KIBANA_SERVICE_HOST:5601}'
    setup.dashboards:
      enabled: true
      retry:
        enabled: true
    setup.template:
      enabled: true
      retry:
        enabled: true
    log.level: debug
kind: ConfigMap
6

It should be logging.level, not log.level. Try that.

7

Doh! Thank you, need to clear the weekend cobwebs out. I got it now...

❯ k logs metricbeat-metricbeat-6tswd
2020-01-13T18:33:57.316Z	INFO	instance/beat.go:610	Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]
2020-01-13T18:33:57.316Z	DEBUG	[beat]	instance/beat.go:662	Beat metadata path: /usr/share/metricbeat/data/meta.json
2020-01-13T18:33:57.316Z	INFO	instance/beat.go:618	Beat ID: 82ff3a1b-a4cd-4e5a-85ef-245a57005b89
2020-01-13T18:33:57.317Z	INFO	[api]	api/server.go:62	Starting stats endpoint
2020-01-13T18:33:57.317Z	DEBUG	[seccomp]	seccomp/seccomp.go:117	Loading syscall filter	{"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchown","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}}
2020-01-13T18:33:57.317Z	INFO	[api]	api/server.go:64	Metrics endpoint listening on: 127.0.0.1:5066 (configured: localhost)
2020-01-13T18:33:57.318Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2020-01-13T18:33:57.318Z	INFO	[beat]	instance/beat.go:941	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat", "data": "/usr/share/metricbeat/data", "home": "/usr/share/metricbeat", "logs": "/usr/share/metricbeat/logs"}, "type": "metricbeat", "uuid": "82ff3a1b-a4cd-4e5a-85ef-245a57005b89"}}}
2020-01-13T18:33:57.318Z	INFO	[beat]	instance/beat.go:950	Build info	{"system_info": {"build": {"commit": "6d0d0ae079e5cb1d4f224801ac6df926dfb1594c", "libbeat": "7.5.0", "time": "2019-11-26T00:09:41.000Z", "version": "7.5.0"}}}
2020-01-13T18:33:57.318Z	INFO	[beat]	instance/beat.go:953	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.12.12"}}}
2020-01-13T18:33:57.319Z	INFO	[beat]	instance/beat.go:957	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-10-23T07:10:56Z","containerized":false,"name":"metricbeat-metricbeat-6tswd","ip":["127.0.0.1/8","172.16.0.83/22"],"kernel_version":"4.15.0-1061-azure","mac":["ca:da:c8:92:2b:e2"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":7,"patch":1908,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-01-13T18:33:57.319Z	INFO	[beat]	instance/beat.go:986	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/metricbeat", "exe": "/usr/share/metricbeat/metricbeat", "name": "metricbeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-01-13T18:33:56.380Z"}}}
2020-01-13T18:33:57.319Z	INFO	instance/beat.go:297	Setup Beat: metricbeat; Version: 7.5.0
2020-01-13T18:33:57.319Z	DEBUG	[beat]	instance/beat.go:323	Initializing output plugins
2020-01-13T18:33:57.319Z	INFO	[index-management]	idxmgmt/std.go:182	Set output.elasticsearch.index to 'metricbeat-7.5.0' as ILM is enabled.
2020-01-13T18:33:57.320Z	INFO	elasticsearch/client.go:171	Elasticsearch url: http://elasticsearch-master:9200
2020-01-13T18:33:57.320Z	DEBUG	[publisher]	pipeline/consumer.go:137	start pipeline event consumer
2020-01-13T18:33:57.321Z	INFO	[publisher]	pipeline/module.go:97	Beat name: metricbeat-metricbeat-6tswd
2020-01-13T18:33:57.329Z	DEBUG	[modules]	beater/metricbeat.go:121	Available modules and metricsets: Register [ModuleFactory:[aws, azure, beat, docker, elasticsearch, kibana, logstash, mongodb, mssql, mysql, oracle, postgresql, system, uwsgi], MetricSetFactory:[aerospike/namespace, apache/status, appsearch/stats, aws/cloudwatch, aws/ec2, aws/rds, aws/s3_daily_storage, aws/s3_request, aws/sqs, azure/compute_vm, azure/compute_vm_scaleset, azure/monitor, beat/state, beat/stats, ceph/cluster_disk, ceph/cluster_health, ceph/cluster_status, ceph/monitor_health, ceph/osd_df, ceph/osd_tree, ceph/pool_disk, consul/agent, coredns/stats, couchbase/bucket, couchbase/cluster, couchbase/node, couchdb/server, docker/container, docker/cpu, docker/diskio, docker/event, docker/healthcheck, docker/image, docker/info, docker/memory, docker/network, dropwizard/collector, elasticsearch/ccr, elasticsearch/cluster_stats, elasticsearch/enrich, elasticsearch/index, elasticsearch/index_recovery, elasticsearch/index_summary, elasticsearch/ml_job, elasticsearch/node, elasticsearch/node_stats, elasticsearch/pending_tasks, elasticsearch/shard, envoyproxy/server, etcd/leader, etcd/metrics, etcd/self, etcd/store, golang/expvar, golang/heap, graphite/server, haproxy/info, haproxy/stat, http/json, http/server, jolokia/jmx, kafka/consumergroup, kafka/partition, kibana/stats, kibana/status, kubernetes/apiserver, kubernetes/container, kubernetes/controllermanager, kubernetes/event, kubernetes/node, kubernetes/pod, kubernetes/proxy, kubernetes/scheduler, kubernetes/state_container, kubernetes/state_cronjob, kubernetes/state_deployment, kubernetes/state_node, kubernetes/state_pod, kubernetes/state_replicaset, kubernetes/state_resourcequota, kubernetes/state_statefulset, kubernetes/system, kubernetes/volume, kvm/dommemstat, logstash/node, logstash/node_stats, memcached/stats, mongodb/collstats, mongodb/dbstats, mongodb/metrics, mongodb/replstatus, mongodb/status, mssql/performance, mssql/transaction_log, munin/node, mysql/galera_status, mysql/status, nats/connections, nats/routes, nats/stats, nats/subscriptions, nginx/stubstatus, oracle/performance, oracle/tablespace, php_fpm/pool, php_fpm/process, postgresql/activity, postgresql/bgwriter, postgresql/database, postgresql/statement, prometheus/collector, rabbitmq/connection, rabbitmq/exchange, rabbitmq/node, rabbitmq/queue, redis/info, redis/key, redis/keyspace, statsd/server, system/core, system/cpu, system/diskio, system/entropy, system/filesystem, system/fsstat, system/load, system/memory, system/network, system/process, system/process_summary, system/raid, system/socket, system/socket_summary, system/uptime, traefik/health, uwsgi/status, vsphere/datastore, vsphere/host, vsphere/virtualmachine, zookeeper/connection, zookeeper/mntr, zookeeper/server], LightModules:[cockroachdb/status, tomcat/threading, tomcat/memory, tomcat/requests, tomcat/cache, aws/elb, aws/ebs]]
2020-01-13T18:33:57.362Z	INFO	add_kubernetes_metadata/kubernetes.go:68	add_kubernetes_metadata: kubernetes env detected, with version: v1.14.6
2020-01-13T18:33:57.363Z	INFO	kubernetes/util.go:94	kubernetes: Using pod name metricbeat-metricbeat-6tswd and namespace elk to discover kubernetes node
2020-01-13T18:33:57.376Z	INFO	kubernetes/util.go:100	kubernetes: Using node aks-agentpool-11078474-2 discovered by in cluster pod node query
2020-01-13T18:33:57.376Z	DEBUG	[kubernetes]	add_kubernetes_metadata/kubernetes.go:138	Initializing a new Kubernetes watcher using host: aks-agentpool-11078474-2
2020-01-13T18:33:57.477Z	DEBUG	[kubernetes]	kubernetes/watcher.go:241	cache sync done
2020-01-13T18:33:57.478Z	DEBUG	[kubernetes]	add_kubernetes_metadata/kubernetes.go:156	add_kubernetes_metadata: adding pod: lendio/lendio-offers-redis-master-0
2020-01-13T18:33:57.478Z	DEBUG	[filters]	add_cloud_metadata/providers.go:126	add_cloud_metadata: starting to fetch metadata, timeout=3s
2020-01-13T18:33:57.478Z	DEBUG	[kubernetes]	add_kubernetes_metadata/kubernetes.go:156	add_kubernetes_metadata: adding pod: monitoring/prom-mfcloud-prometheus-node-exporter-7s689
2020-01-13T18:33:57.478Z	DEBUG	[kubernetes]	add_kubernetes_metadata/kubernetes.go:156	add_kubernetes_metadata: adding pod: linkerd/linkerd-grafana-5d57b66477-55xd7
2020-01-13T18:33:57.478Z	DEBUG	[kubernetes]	add_kubernetes_metadata/kubernetes.go:156	add_kubernetes_metadata: adding pod: linkerd/linkerd-controller-79d9f789b6-p67b5
2020-01-13T18:33:57.478Z	DEBUG	[kubernetes]	add_kubernetes_metadata/kubernetes.go:156	add_kubernetes_metadata: adding pod: cert-manager/cert-manager-756d9f56d6-dnfz4
2020-01-13T18:33:57.478Z	DEBUG	[kubernetes]	add_kubernetes_metadata/kubernetes.go:156	add_kubernetes_metadata: adding pod: default/mf-tickets-form-5664c88f9d-r7xcx
2020-01-13T18:33:57.479Z	DEBUG	[kubernetes]	add_kubernetes_metadata/kubernetes.go:156	add_kubernetes_metadata: adding pod: kube-system/dash-kubernetes-dashboard-5649c959cd-2g5pz
8 
2020-01-13T18:33:57.485Z	DEBUG	[filters]	add_cloud_metadata/providers.go:162	add_cloud_metadata: received disposition for gcp after 7.266946ms. result=[provider:gcp, error=failed with http status code 404, metadata={}]
2020-01-13T18:33:57.486Z	DEBUG	[filters]	add_cloud_metadata/providers.go:162	add_cloud_metadata: received disposition for openstack after 8.252652ms. result=[provider:openstack, error=failed with http status code 404, metadata={}]
2020-01-13T18:33:57.487Z	DEBUG	[filters]	add_cloud_metadata/providers.go:162	add_cloud_metadata: received disposition for aws after 9.293959ms. result=[provider:aws, error=failed with http status code 404, metadata={}]
2020-01-13T18:33:57.488Z	DEBUG	[filters]	add_cloud_metadata/providers.go:162	add_cloud_metadata: received disposition for az after 10.748468ms. result=[provider:az, error=<nil>, metadata={"instance":{"id":"e280112c-ff80-4da5-bf45-8fb1e64ccce5","name":"aks-agentpool-11078474-2"},"machine":{"type":"Standard_D2s_v3"},"provider":"az","region":"westus"}]
2020-01-13T18:33:57.489Z	DEBUG	[filters]	add_cloud_metadata/providers.go:129	add_cloud_metadata: fetchMetadata ran for 10.797568ms
2020-01-13T18:33:57.489Z	INFO	add_cloud_metadata/add_cloud_metadata.go:93	add_cloud_metadata: hosting provider type detected as az, metadata={"instance":{"id":"e280112c-ff80-4da5-bf45-8fb1e64ccce5","name":"aks-agentpool-11078474-2"},"machine":{"type":"Standard_D2s_v3"},"provider":"az","region":"westus"}
2020-01-13T18:33:57.489Z	DEBUG	[processors]	processors/processor.go:101	Generated new processors: add_kubernetes_metadata, add_cloud_metadata={"instance":{"id":"e280112c-ff80-4da5-bf45-8fb1e64ccce5","name":"aks-agentpool-11078474-2"},"machine":{"type":"Standard_D2s_v3"},"provider":"az","region":"westus"}
2020-01-13T18:33:57.490Z	INFO	kubernetes/util.go:79	kubernetes: Using node aks-agentpool-11078474-2 provided in the config
2020-01-13T18:33:57.490Z	DEBUG	[kubernetes]	util/kubernetes.go:95	Initializing a new Kubernetes watcher using host: aks-agentpool-11078474-2
2020-01-13T18:33:57.492Z	DEBUG	[kubernetes]	util/kubernetes.go:95	Initializing a new Kubernetes watcher using host: aks-agentpool-11078474-2
2020-01-13T18:33:57.495Z	INFO	kubernetes/util.go:79	kubernetes: Using node aks-agentpool-11078474-2 provided in the config
2020-01-13T18:33:57.495Z	DEBUG	[kubernetes]	util/kubernetes.go:95	Initializing a new Kubernetes watcher using host: aks-agentpool-11078474-2
2020-01-13T18:33:57.501Z	DEBUG	[kubernetes]	util/kubernetes.go:95	Initializing a new Kubernetes watcher using host:
2020-01-13T18:33:57.503Z	DEBUG	[kubernetes]	util/kubernetes.go:95	Initializing a new Kubernetes watcher using host:
2020-01-13T18:33:57.509Z	DEBUG	[kubernetes]	util/kubernetes.go:95	Initializing a new Kubernetes watcher using host:
2020-01-13T18:33:57.510Z	DEBUG	[kubernetes]	util/kubernetes.go:95	Initializing a new Kubernetes watcher using host:
2020-01-13T18:33:57.511Z	DEBUG	[kubernetes]	util/kubernetes.go:95	Initializing a new Kubernetes watcher using host:
2020-01-13T18:33:57.512Z	DEBUG	[system.process]	process/process.go:84	process cgroup data collection is enabled, using hostfs=''
2020-01-13T18:33:57.516Z	DEBUG	[conditions]	conditions/conditions.go:98	New condition regexp: map[]
2020-01-13T18:33:57.516Z	DEBUG	[processors]	processors/processor.go:101	Generated new processors: drop_event, condition=regexp: map[]
2020-01-13T18:33:57.516Z	INFO	filesystem/filesystem.go:57	Ignoring filesystem types: sysfs, rootfs, ramfs, bdev, proc, cpuset, cgroup, cgroup2, tmpfs, devtmpfs, configfs, debugfs, tracefs, securityfs, sockfs, dax, bpf, pipefs, hugetlbfs, devpts, ecryptfs, fuse, fusectl, pstore, mqueue, autofs, rpc_pipefs, aufs, overlay, cifs
2020-01-13T18:33:57.516Z	INFO	[system.fsstat]	fsstat/fsstat.go:56	Ignoring filesystem types: %ssysfs, rootfs, ramfs, bdev, proc, cpuset, cgroup, cgroup2, tmpfs, devtmpfs, configfs, debugfs, tracefs, securityfs, sockfs, dax, bpf, pipefs, hugetlbfs, devpts, ecryptfs, fuse, fusectl, pstore, mqueue, autofs, rpc_pipefs, aufs, overlay, cifs
2020-01-13T18:33:57.516Z	WARN	[cfgwarn]	kubernetes/kubernetes.go:59	BETA: The kubernetes autodiscover is beta
2020-01-13T18:33:57.517Z	INFO	kubernetes/util.go:79	kubernetes: Using node metricbeat-metricbeat-6tswd provided in the config
2020-01-13T18:33:57.517Z	DEBUG	[autodiscover]	kubernetes/kubernetes.go:84	Initializing a new Kubernetes watcher using host: metricbeat-metricbeat-6tswd
2020-01-13T18:33:57.517Z	DEBUG	[conditions]	conditions/conditions.go:98	New condition contains: map[]
2020-01-13T18:33:57.517Z	DEBUG	[conditions]	conditions/conditions.go:98	New condition contains: map[]
2020-01-13T18:33:57.517Z	DEBUG	[autodiscover]	autodiscover/autodiscover.go:82	Configured autodiscover provider: kubernetes
2020-01-13T18:33:57.518Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2020-01-13T18:33:57.518Z	INFO	kibana/client.go:117	Kibana url: http://10.0.252.214:5601
2020-01-13T18:33:57.650Z	DEBUG	[kubernetes]	add_kubernetes_metadata/kubernetes.go:161	add_kubernetes_metadata: updating pod: elk/metricbeat-metricbeat-6tswd
2020-01-13T18:33:58.092Z	INFO	kibana/client.go:117	Kibana url: http://10.0.252.214:5601
2020-01-13T18:33:58.115Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Initialize the Kibana 7.5.0 loader
2020-01-13T18:33:58.115Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Kibana URL http://10.0.252.214:5601
2020-01-13T18:33:58.115Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Importing directory /usr/share/metricbeat/kibana/7
2020-01-13T18:33:59.302Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import directory /usr/share/metricbeat/kibana/7
2020-01-13T18:33:59.303Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-aerospike-overview.json
2020-01-13T18:34:00.286Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-apache-overview.json
2020-01-13T18:34:01.305Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-aws-ebs-overview.json
2020-01-13T18:34:02.327Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-aws-ec2-overview.json
2020-01-13T18:34:03.365Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-aws-elb-overview.json
2020-01-13T18:34:04.385Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-aws-overview.json
2020-01-13T18:34:05.414Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-aws-rds-overview.json
2020-01-13T18:34:06.432Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-aws-s3-overview.json
2020-01-13T18:34:07.446Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-aws-sqs-overview.json
2020-01-13T18:34:08.463Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-azure-vm-guestmetrics-overview.json
2020-01-13T18:34:09.475Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-azure-vm-overview.json
2020-01-13T18:34:10.590Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-azure-vmss-overview.json
2020-01-13T18:34:11.517Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-ceph-overview.json
2020-01-13T18:34:12.541Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-cockroachdb-overview.json
2020-01-13T18:34:12.794Z	DEBUG	[kubernetes]	add_kubernetes_metadata/kubernetes.go:161	add_kubernetes_metadata: updating pod: elk/metricbeat-metricbeat-6tswd
2020-01-13T18:34:13.554Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-consul-overview.json
2020-01-13T18:34:14.578Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-containers-overview.json
2020-01-13T18:34:15.601Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-coredns-overview.json
2020-01-13T18:34:16.604Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-couchbase-overview.json
2020-01-13T18:34:17.616Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-couchdb-overview.json
2020-01-13T18:34:18.640Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-docker-overview.json
2020-01-13T18:34:19.813Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-golang-overview.json
2020-01-13T18:34:20.829Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-haproxy-backend.json
2020-01-13T18:34:21.914Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-haproxy-frontend.json
2020-01-13T18:34:22.925Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-haproxy-http-backend.json
2020-01-13T18:34:23.957Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-haproxy-http-frontend.json
2020-01-13T18:34:24.965Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-haproxy-http-server.json
2020-01-13T18:34:25.989Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-haproxy-overview.json
2020-01-13T18:34:27.000Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:146	Import dashboard from /usr/share/metricbeat/kibana/7/dashboard/Metricbeat-haproxy-visualizations.json
2020-01-13T18:34:27.523Z	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":80,"time":{"ms":89}},"total":{"ticks":930,"time":{"ms":943},"value":930},"user":{"ticks":850,"time":{"ms":854}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":11},"info":{"ephemeral_id":"0a6e8e9c-4a71-4ccc-b9de-e70af97876ed","uptime":{"ms":30259}},"memstats":{"gc_next":19046352,"memory_alloc":10719680,"memory_total":100758840,"rss":97288192},"runtime":{"goroutines":54}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":2},"load":{"1":2.53,"15":2.19,"5":2.8,"norm":{"1":1.265,"15":1.095,"5":1.4}}}}}}
9

Hmm, I'm not seeing anything particularly useful in these logs so far. Any chance you could post the next 30 seconds or so worth of logs as well? Thanks!

10

Here you go!

2020-01-13T18:46:57.283Z        DEBUG   [processors]    processing/processors.go:186    Publish event: {
  "@timestamp": "2020-01-13T18:46:57.182Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.5.0"
  },
  "metricset": {
    "period": 10000,
    "name": "state_pod"
  },
  "service": {
    "address": "metricbeat-kube-state-metrics:8080",
    "type": "kubernetes"
  },
  "kubernetes": {
    "pod": {
      "host_ip": "172.16.0.128",
      "ip": "172.16.0.128",
      "uid": "a3d54b98-1d3f-11ea-b8db-721a07af99f9",
      "name": "azure-cni-networkmonitor-sh586",
      "status": {
        "ready": "true",
        "scheduled": "true",
        "phase": "running"
      }
    },
    "namespace": "kube-system",
    "node": {
      "name": "aks-agentpool-11078474-4"
    },
    "labels": {
      "component": "azure-cni-networkmonitor",
      "controller-revision-hash": "84884bc6d9",
      "pod-template-generation": "2",
      "tier": "node"
    }
  },
  "ecs": {
    "version": "1.1.0"
  },
  "host": {
    "name": "metricbeat-metricbeat-6tswd"
  },
  "agent": {
    "id": "82ff3a1b-a4cd-4e5a-85ef-245a57005b89",
    "version": "7.5.0",
    "type": "metricbeat",
    "ephemeral_id": "0a6e8e9c-4a71-4ccc-b9de-e70af97876ed",
    "hostname": "metricbeat-metricbeat-6tswd"
  },
  "event": {
    "duration": 100522751,
    "dataset": "kubernetes.pod",
    "module": "kubernetes"
2020-01-13T18:46:57.283Z        DEBUG   [processors]    processing/processors.go:186    Publish event: {
  "@timestamp": "2020-01-13T18:46:57.182Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.5.0"
  },
  "host": {
    "name": "metricbeat-metricbeat-6tswd"
  },
  "agent": {
    "ephemeral_id": "0a6e8e9c-4a71-4ccc-b9de-e70af97876ed",
    "hostname": "metricbeat-metricbeat-6tswd",
    "id": "82ff3a1b-a4cd-4e5a-85ef-245a57005b89",
    "version": "7.5.0",
    "type": "metricbeat"
  },
  "event": {
    "dataset": "kubernetes.pod",
    "module": "kubernetes",
    "duration": 100537151
  },
  "metricset": {
    "name": "state_pod",
    "period": 10000
  },
  "service": {
    "address": "metricbeat-kube-state-metrics:8080",
    "type": "kubernetes"
  },
  "kubernetes": {
    "namespace": "haliotis",
    "node": {
      "name": "aks-agentpool-11078474-1"
    },
    "labels": {
      "controller-uid": "699157ca-3286-11ea-b8db-721a07af99f9",
      "job-name": "haliotis-daily-dnb-file-download-1578536340"
    },
    "pod": {
      "uid": "69a1de3f-3286-11ea-b8db-721a07af99f9",
      "name": "haliotis-daily-dnb-file-download-1578536340-jgqd5",
      "status": {
        "scheduled": "true",
        "phase": "pending",
        "ready": "false"
      },
      "host_ip": "172.16.0.4",
      "ip": "172.16.0.8"
    }
  },
  "ecs": {
    "version": "1.1.0"
  }
}
"kubernetes": {
    "namespace": "kube-system",
    "node": {
      "name": "aks-agentpool-11078474-1"
    },
    "replicaset": {
      "name": "metrics-server-58b6fcfd54"
    },
    "labels": {
      "k8s-app": "metrics-server",
      "pod-template-hash": "58b6fcfd54"
    },
    "pod": {
      "name": "metrics-server-58b6fcfd54-8q8pj",
      "status": {
        "scheduled": "true",
        "phase": "running",
        "ready": "true"
      },
      "host_ip": "172.16.0.4",
      "ip": "172.16.0.25",
      "uid": "b68d65d1-32ab-11ea-b8db-721a07af99f9"
    }
  },
  "event": {
    "module": "kubernetes",
    "duration": 100542051,
    "dataset": "kubernetes.pod"
  },
  "ecs": {
    "version": "1.1.0"
  },
  "host": {
    "name": "metricbeat-metricbeat-6tswd"
  },
  "agent": {
    "ephemeral_id": "0a6e8e9c-4a71-4ccc-b9de-e70af97876ed",
    "hostname": "metricbeat-metricbeat-6tswd",
    "id": "82ff3a1b-a4cd-4e5a-85ef-245a57005b89",
    "version": "7.5.0",
    "type": "metricbeat"
  },
  "metricset": {
    "name": "state_pod",
    "period": 10000
  }
}
2020-01-13T18:46:57.283Z        DEBUG   [processors]    processing/processors.go:186    Publish event: {
  "@timestamp": "2020-01-13T18:46:57.182Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.5.0"
  },
  "host": {
    "name": "metricbeat-metricbeat-6tswd"
  },
  "agent": {
    "ephemeral_id": "0a6e8e9c-4a71-4ccc-b9de-e70af97876ed",
    "hostname": "metricbeat-metricbeat-6tswd",
    "id": "82ff3a1b-a4cd-4e5a-85ef-245a57005b89",
    "version": "7.5.0",
    "type": "metricbeat"
  },
11 
2020-01-13T18:50:41.089Z	DEBUG	[processes]	process/process.go:434	Filtered top processes down to 1 processes
2020-01-13T18:50:41.090Z	DEBUG	[processors]	processing/processors.go:186	Publish event: {
  "@timestamp": "2020-01-13T18:50:41.088Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.5.0"
  },
  "process": {
    "pgid": 1,
    "working_directory": "/usr/share/metricbeat",
    "executable": "/usr/share/metricbeat/metricbeat",
    "args": [
      "metricbeat",
      "-e",
      "-E",
      "http.enabled=true"
    ],
    "name": "metricbeat",
    "pid": 1,
    "ppid": 0
  },
  "user": {
    "name": "root"
  },
  "metricset": {
    "period": 10000,
    "name": "process"
  },
  "service": {
    "type": "system"
  },
  "ecs": {
    "version": "1.1.0"
  },
  "event": {
    "dataset": "system.process",
    "module": "system",
    "duration": 1151107
  },
  "system": {
    "process": {
      "fd": {
        "open": 21,
        "limit": {
          "soft": 1048576,
          "hard": 1048576
        }
      },
      "cpu": {
        "total": {
          "value": 114030,
          "pct": 0.12,
          "norm": {
            "pct": 0.06
          }
        },
        "start_time": "2020-01-13T18:33:56.000Z"
      },
      "memory": {
        "share": 53952512,
        "size": 600961024,
        "rss": {
          "pct": 0.0209,
          "bytes": 174215168
        }
      },
      "cgroup": {
        "path": "/kubepods/burstable/pod40ce4080-3633-11ea-b8db-721a07af99f9/4f8d718cfc6313fa94d0743afe9e385c109ac326d26bd21ea243af6572d6137d",
        "cpu": {
          "cfs": {
            "period": {
              "us": 0
            },
            "quota": {
              "us": 0
            },
            "shares": 0
          },
          "rt": {
            "runtime": {
              "us": 0
            },
            "period": {
              "us": 0
            }
          },
          "stats": {
            "throttled": {
              "ns": 0,
              "periods": 0
            },
            "periods": 0
          },
          "id": "4f8d718cfc6313fa94d0743afe9e385c109ac326d26bd21ea243af6572d6137d",
          "path": "/kubepods/burstable/pod40ce4080-3633-11ea-b8db-721a07af99f9/4f8d718cfc6313fa94d0743afe9e385c109ac326d26bd21ea243af6572d6137d"
        },
        "cpuacct": {
          "total": {
            "ns": 0
          },
          "stats": {
            "user": {
              "ns": 0
            },
            "system": {
              "ns": 0
            }
          },
          "percpu": {},
          "id": "4f8d718cfc6313fa94d0743afe9e385c109ac326d26bd21ea243af6572d6137d",
          "path": "/kubepods/burstable/pod40ce4080-3633-11ea-b8db-721a07af99f9/4f8d718cfc6313fa94d0743afe9e385c109ac326d26bd21ea243af6572d6137d"
        },
        "memory": {
          "kmem": {
            "failures": 0,
            "limit": {
              "bytes": 0
            },
            "usage": {
              "max": {
                "bytes": 0
              },
              "bytes": 0
            }
          },
          "kmem_tcp": {
            "limit": {
              "bytes": 0
            },
            "usage": {
              "bytes": 0,
              "max": {
                "bytes": 0
              }
            },
            "failures": 0
          },
          "stats": {
            "rss_huge": {
              "bytes": 0
            },
            "active_anon": {
              "bytes": 0
            },
            "cache": {
              "bytes": 0
            },
            "inactive_anon": {
              "bytes": 0
            },
            "inactive_file": {
              "bytes": 0
            },
            "major_page_faults": 0,
            "swap": {
              "bytes": 0
            },
            "rss": {
              "bytes": 0
            },
            "page_faults": 0,
            "pages_out": 0,
            "active_file": {
              "bytes": 0
            },
            "hierarchical_memsw_limit": {
              "bytes": 0
            },
            "unevictable": {
              "bytes": 0
            },
            "hierarchical_memory_limit": {
              "bytes": 0
            },
            "pages_in": 0,
            "mapped_file": {
              "bytes": 0
            }
          },
          "id": "4f8d718cfc6313fa94d0743afe9e385c109ac326d26bd21ea243af6572d6137d",
          "path": "/kubepods/burstable/pod40ce4080-3633-11ea-b8db-721a07af99f9/4f8d718cfc6313fa94d0743afe9e385c109ac326d26bd21ea243af6572d6137d",
          "mem": {
            "failures": 0,
            "limit": {
              "bytes": 0
            },
            "usage": {
              "bytes": 0,
              "max": {
                "bytes": 0
              }
            }
          },
          "memsw": {
            "failures": 0,
            "limit": {
              "bytes": 0
            },
            "usage": {
              "bytes": 0,
              "max": {
                "bytes": 0
              }
            }
          }
        },
        "blkio": {
          "total": {
            "ios": 0,
            "bytes": 0
          },
          "id": "4f8d718cfc6313fa94d0743afe9e385c109ac326d26bd21ea243af6572d6137d",
          "path": "/kubepods/burstable/pod40ce4080-3633-11ea-b8db-721a07af99f9/4f8d718cfc6313fa94d0743afe9e385c109ac326d26bd21ea243af6572d6137d"
        },
        "id": "4f8d718cfc6313fa94d0743afe9e385c109ac326d26bd21ea243af6572d6137d"
      },
      "state": "sleeping",
      "cmdline": "metricbeat -e -E http.enabled=true"
    }
  },
  "host": {
    "name": "metricbeat-metricbeat-6tswd"
  },
  "agent": {
    "version": "7.5.0",
    "type": "metricbeat",
    "ephemeral_id": "0a6e8e9c-4a71-4ccc-b9de-e70af97876ed",
    "hostname": "metricbeat-metricbeat-6tswd",
    "id": "82ff3a1b-a4cd-4e5a-85ef-245a57005b89"
  }
}
2020-01-13T18:50:41.625Z	DEBUG	[processors]	processing/processors.go:186	Publish event: {
  "@timestamp": "2020-01-13T18:50:41.625Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.5.0"
  },
  "system": {
    "load": {
      "norm": {
        "1": 1.23,
        "5": 0.9,
        "15": 0.835
      },
      "cores": 2,
      "1": 2.46,
      "5": 1.8,
      "15": 1.67
    }
  },
  "event": {
    "duration": 124801,
    "dataset": "system.load",
    "module": "system"
  },
  "metricset": {
    "name": "load",
    "period": 10000
  },
  "service": {
    "type": "system"
  },
  "ecs": {
    "version": "1.1.0"
  },
  "host": {
    "name": "metricbeat-metricbeat-6tswd"
  },
  "agent": {
    "ephemeral_id": "0a6e8e9c-4a71-4ccc-b9de-e70af97876ed",
    "hostname": "metricbeat-metricbeat-6tswd",
    "id": "82ff3a1b-a4cd-4e5a-85ef-245a57005b89",
    "version": "7.5.0",
    "type": "metricbeat"
  }
}
12

@shaunak Any clues here?

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.