No data sent to configured output

[code-chris (Christian Kotzbauer)]

Hi,
I created a Metricbeat instance with Docker on Linux, but Metricbeat does not emit any error messages, metric results or something else. Only the internal metric-message "INFO Non-zero metrics in the last 30s" is shown. The "-e" flag for console output is set.

I tried to output the results to logstash, directly to elasticsearch or the the console, but it doesn't work.

The only thing metricbeat has done was the creation of the kibana dashboards. The metrics "docker", "http", "nginx" and "system" are configured in the "modules.d." directory.

Docker: 17.09-ce
Elastic-Stack: 6.1.0
Docker-Host: Ubuntu 17.10

My metricbeat.yml:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml

setup.kibana.host: "kibana:5601"
setup.dashboards.enabled: true

#output.logstash:
#  hosts: ["logstash:5044"]

#output.elasticsearch:
#  hosts: ["http://elasticsearch:9200"]

output.console:
  pretty: true

Maybe someone knows what's going wrong here :). Thanks!

[ruflin]

Can you run metricbeat modules list to see which modules are enabled? Also you can run metricbeat with log level debug and share the output here?

[code-chris (Christian Kotzbauer)]

Module list:

$ metricbeat modules list

Enabled:
docker
http
nginx
system

Disabled:
aerospike
apache
ceph
couchbase
dropwizard
elasticsearch
etcd
golang
graphite
haproxy
jolokia
kafka
kibana
kubernetes
logstash
memcached
mongodb
mysql
php_fpm
postgresql
prometheus
rabbitmq
redis
vsphere
windows
zookeeper

[code-chris (Christian Kotzbauer)]

Debug log after startup (Part 1):

2017/12/28 07:42:21.710325 beat.go:436: INFO Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]
2017/12/28 07:42:21.710373 beat.go:463: DBG [beat] Beat metadata path: /usr/share/metricbeat/data/meta.json
2017/12/28 07:42:21.710488 metrics.go:23: INFO Metrics logging every 30s
2017/12/28 07:42:21.710493 beat.go:443: INFO Beat UUID: fe851e6c-8e74-44d8-b116-c5f566ecbbaa
2017/12/28 07:42:21.710578 beat.go:203: INFO Setup Beat: metricbeat; Version: 6.1.0
2017/12/28 07:42:21.710621 beat.go:215: DBG [beat] Initializing output plugins
2017/12/28 07:42:21.710649 processor.go:49: DBG [processors] Processors:
2017/12/28 07:42:21.711048 logger.go:18: DBG [publish] start pipeline event consumer
2017/12/28 07:42:21.711076 module.go:76: INFO Beat name: 9645233a7ed3
2017/12/28 07:42:21.711217 metricbeat.go:39: DBG [modules] Register [ModuleFactory:[docker, mongodb, mysql, postgresql, system], MetricSetFactory:[aerospike/namespace, apache/status, ceph/cluster_disk, ceph/cluster_health, ceph/cluster_status, ceph/monitor_health, ceph/osd_tree, ceph/pool_disk, couchbase/bucket, couchbase/cluster, couchbase/node, docker/container, docker/cpu, docker/diskio, docker/healthcheck, docker/image, docker/info, docker/memory, docker/network, dropwizard/collector, elasticsearch/node, elasticsearch/node_stats, etcd/leader, etcd/self, etcd/store, golang/expvar, golang/heap, graphite/server, haproxy/info, haproxy/stat, http/json, http/server, jolokia/jmx, kafka/consumergroup, kafka/partition, kibana/status, kubernetes/container, kubernetes/event, kubernetes/node, kubernetes/pod, kubernetes/state_container, kubernetes/state_deployment, kubernetes/state_node, kubernetes/state_pod, kubernetes/state_replicaset, kubernetes/system, kubernetes/volume, logstash/node, logstash/node_stats, memcached/stats, mongodb/dbstats, mongodb/status, mysql/status, nginx/stubstatus, php_fpm/pool, postgresql/activity, postgresql/bgwriter, postgresql/database, prometheus/collector, prometheus/stats, rabbitmq/node, rabbitmq/queue, redis/info, redis/keyspace, system/core, system/cpu, system/diskio, system/filesystem, system/fsstat, system/load, system/memory, system/network, system/process, system/process_summary, system/socket, system/uptime, vsphere/datastore, vsphere/host, vsphere/virtualmachine, zookeeper/mntr]]
2017/12/28 07:42:21.711495 client.go:69: INFO Kibana url: http://kibana:5601
2017/12/28 07:42:21.722161 kibana_loader.go:103: DBG [dashboards] Initialize the Kibana 6.1.0 loader
2017/12/28 07:42:21.722258 kibana_loader.go:103: DBG [dashboards] Kibana URL http://kibana:5601
2017/12/28 07:42:21.722330 kibana_loader.go:103: DBG [dashboards] Importing directory /usr/share/metricbeat/kibana/default
2017/12/28 07:42:21.722739 kibana_loader.go:103: DBG [dashboards] Import directory /usr/share/metricbeat/kibana/default
2017/12/28 07:42:21.723046 kibana_loader.go:103: DBG [dashboards] Import index-pattern from /usr/share/metricbeat/kibana/default/index-pattern/metricbeat.json
2017/12/28 07:42:22.547668 kibana_loader.go:103: DBG [dashboards] Import directory /usr/share/metricbeat/kibana/default
2017/12/28 07:42:22.548175 kibana_loader.go:103: DBG [dashboards] Import dashboard from /usr/share/metricbeat/kibana/default/dashboard/Metricbeat-apache-overview.json
2017/12/28 07:42:23.555664 kibana_loader.go:103: DBG [dashboards] Import dashboard from /usr/share/metricbeat/kibana/default/dashboard/Metricbeat-containers-overview.json
2017/12/28 07:42:24.567745 kibana_loader.go:103: DBG [dashboards] Import dashboard from /usr/share/metricbeat/kibana/default/dashboard/Metricbeat-docker-overview.json
2017/12/28 07:42:25.583706 kibana_loader.go:103: DBG [dashboards] Import dashboard from /usr/share/metricbeat/kibana/default/dashboard/Metricbeat-golang-overview.json
2017/12/28 07:42:26.587673 kibana_loader.go:103: DBG [dashboards] Import dashboard from /usr/share/metricbeat/kibana/default/dashboard/Metricbeat-host-overview.json
2017/12/28 07:42:27.562914 kibana_loader.go:103: DBG [dashboards] Import dashboard from /usr/share/metricbeat/kibana/default/dashboard/Metricbeat-kubernetes-overview.json
2017/12/28 07:42:28.619561 kibana_loader.go:103: DBG [dashboards] Import dashboard from /usr/share/metricbeat/kibana/default/dashboard/Metricbeat-mongodb-overview.json
2017/12/28 07:42:29.635615 kibana_loader.go:103: DBG [dashboards] Import dashboard from /usr/share/metricbeat/kibana/default/dashboard/Metricbeat-mysql-overview.json
2017/12/28 07:42:30.651624 kibana_loader.go:103: DBG [dashboards] Import dashboard from /usr/share/metricbeat/kibana/default/dashboard/Metricbeat-rabbitmq-overview.json
2017/12/28 07:42:31.659660 kibana_loader.go:103: DBG [dashboards] Import dashboard from /usr/share/metricbeat/kibana/default/dashboard/Metricbeat-redis-overview.json
2017/12/28 07:42:32.671584 kibana_loader.go:103: DBG [dashboards] Import dashboard from /usr/share/metricbeat/kibana/default/dashboard/Metricbeat-system-overview.json
2017/12/28 07:42:33.687744 kibana_loader.go:103: DBG [dashboards] Import dashboard from /usr/share/metricbeat/kibana/default/dashboard/metricbeat-windows-service.json
2017/12/28 07:42:34.703825 beat.go:551: INFO Kibana dashboards successfully loaded.
2017/12/28 07:42:34.703882 beat.go:276: INFO metricbeat start running.

[code-chris (Christian Kotzbauer)]

Debug log after startup (Part 2):

2017/12/28 07:42:34.704101 reload.go:95: DBG [cfgfile] Checking module configs from: /usr/share/metricbeat/modules.d/*.yml
2017/12/28 07:42:34.705453 cfgfile.go:143: DBG [cfgfile] Load config from file: /usr/share/metricbeat/modules.d/docker.yml
2017/12/28 07:42:34.706180 cfgfile.go:143: DBG [cfgfile] Load config from file: /usr/share/metricbeat/modules.d/http.yml
2017/12/28 07:42:34.706834 cfgfile.go:143: DBG [cfgfile] Load config from file: /usr/share/metricbeat/modules.d/nginx.yml
2017/12/28 07:42:34.707487 cfgfile.go:143: DBG [cfgfile] Load config from file: /usr/share/metricbeat/modules.d/system.yml
2017/12/28 07:42:34.708359 reload.go:109: DBG [cfgfile] Number of module configs found: 0
2017/12/28 07:42:34.708434 reload.go:127: INFO Config reloader started
2017/12/28 07:42:34.708615 reload.go:151: DBG [cfgfile] Scan for new config files
2017/12/28 07:42:34.709196 cfgfile.go:143: DBG [cfgfile] Load config from file: /usr/share/metricbeat/modules.d/docker.yml
2017/12/28 07:42:34.709847 cfgfile.go:143: DBG [cfgfile] Load config from file: /usr/share/metricbeat/modules.d/http.yml
2017/12/28 07:42:34.710445 cfgfile.go:143: DBG [cfgfile] Load config from file: /usr/share/metricbeat/modules.d/nginx.yml
2017/12/28 07:42:34.710994 cfgfile.go:143: DBG [cfgfile] Load config from file: /usr/share/metricbeat/modules.d/system.yml
2017/12/28 07:42:34.711725 reload.go:170: DBG [cfgfile] Number of module configs found: 0
2017/12/28 07:42:34.711753 reload.go:219: INFO Loading of config files completed.
2017/12/28 07:42:51.711156 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30019 beat.memstats.gc_next=4194304 beat.memstats.memory_alloc=3179984 beat.memstats.memory_total=10638768 libbeat.config.module.running=0 libbeat.config.reloads=1 libbeat.output.type=console libbeat.pipeline.clients=0 libbeat.pipeline.events.active=0
2017/12/28 07:43:21.710740 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=4194304 beat.memstats.memory_alloc=3188496 beat.memstats.memory_total=10647280 libbeat.config.module.running=0 libbeat.pipeline.clients=0 libbeat.pipeline.events.active=0
2017/12/28 07:43:51.710926 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=4194304 beat.memstats.memory_alloc=3194896 beat.memstats.memory_total=10653680 libbeat.config.module.running=0 libbeat.pipeline.clients=0 libbeat.pipeline.events.active=0

[code-chris (Christian Kotzbauer)]

I found the problem.

All my module configs started with the line "metricbeat.modules:". But this seems to be only neccessary if they are configured directly in metricbeat.yml. I removed the line and all modules are working now.