Metricbeat output not in elasticsearch

Hi, I'm trying to setup a metricbeat on a MacBook for test purposes.
My ELK Stack is running in docker dontainers (docker-elk) which I updated to 7.4.2.

I setup the metricbeat in Kibana, setup the dashboards and the metricbeat index is showing up in the elasticsearch index management.

I can start the metricbeat without any problem on my Mac but it's data won't show up in elasticsearch:

curl -XGET 'http://localhost:9200/metricbeat-*/_search?pretty'

{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : null,
"hits" :
}
}

I tested the metricbeat config, modules and the output. Everything seems to be fine. I'm also just running the system module with the standard config.yml

Here is my metricbeat.yml :

#========================= Central Management =================================

# Beats is configured under central management, you can define most settings
# from the Kibana UI. You can update this file to configure the settings that
# are not supported by Kibana Beats management.

management:
  enabled: true
  period: 1m0s
  events_reporter:
    period: 30s
    max_batch_size: 1000
  access_token: ${management.accesstoken}
  kibana:
    protocol: http
    host: localhost:5601
    ssl: null
    timeout: 10s
    ignoreversion: true
  blacklist:
    output: console|file

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/metricbeat
  name: metricbeat
  keepfiles: 7
  permissions: 0644

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml

#----------------------------- Elasticsearch output -------------------------

output.elasticsearch:
  hosts: ["localhost:9200"]
  username: "elastic"
  password: "changeme"

#----------------------------- Kibana output --------------------------------

setup.kibana:
  host: "localhost:5601"
  username: "elastic"
  password: "changeme"
  dashboards.enabled: false

#----------------------------- Logstash output --------------------------------
# output.logstash:
#   hosts: ["localhost:5044"]
#   username: "elastic"
#   password: "changeme"

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

#============================== X-Pack Monitoring ===============================
# metricbeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

I could really use some help.
Thanks in advance.

Hi @Lieven

What do the Metricbeat logs say? Do you even have authentication enabled? Can you please share your configuration of your ELK Stack?

Please share these informations with us.

Hi @MiTschMR,

the elasticsearch.yml from my docker enviroment looks like this:

## Default Elasticsearch configuration from Elasticsearch base image.
## https://github.com/elastic/elasticsearch/blob/master/distribution/docker/src/docker/config/elasticsearch.yml
#
cluster.name: "docker-cluster"
network.host: 0.0.0.0

## Use single node discovery in order to disable production mode and avoid bootstrap checks
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
#
discovery.type: single-node

## X-Pack settings
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-xpack.html
#
xpack.license.self_generated.type: trial
xpack.security.enabled: true
xpack.monitoring.collection.enabled: true

In terms of logging. There is no logfile in /var/log/metricbreat

I just setup a heartbeat monitor, which is working fine.
Elasticsearch is outputting all the info via cURL and yes authentication seems to be required because cURL won't work without the login credentials.

I guess elasticsearch is just working fine but metricbeat won't ship the data for some reason.

Hi @Lieven

Can you please share the output of the following command according to this site:

./metricbeat modules list 

And also do this:

./metricbeat test config
Enabled:
system

Disabled:
aerospike
apache
aws
beat
beat-xpack
ceph
cockroachdb
consul
coredns
couchbase
couchdb
docker
dropwizard
elasticsearch
elasticsearch-xpack
envoyproxy
etcd
golang
graphite
haproxy
http
jolokia
kafka
kibana
kibana-xpack
kubernetes
kvm
logstash
logstash-xpack
memcached
mongodb
mssql
munin
mysql
nats
nginx
oracle
php_fpm
postgresql
prometheus
rabbitmq
redis
statsd
traefik
uwsgi
vsphere
windows
zookeeper

Config OK

Well I found out why metricbeat isn't outputting a logfile, I just had to start it without the -e tag.

Here is the log:> >

> 2019-11-14T16:17:11.449+0100	INFO	instance/beat.go:292	Setup Beat: metricbeat; Version: 7.4.2
> 2019-11-14T16:17:11.449+0100	INFO	[index-management]	idxmgmt/std.go:178	Set output.elasticsearch.index to 'metricbeat-7.4.2' as ILM is enabled.
> 2019-11-14T16:17:11.449+0100	INFO	elasticsearch/client.go:170	Elasticsearch url: http://localhost:9200
> 2019-11-14T16:17:11.451+0100	INFO	[publisher]	pipeline/module.go:97	Beat name: MacBook
> 2019-11-14T16:17:11.451+0100	INFO	instance/beat.go:422	metricbeat start running.
> 2019-11-14T16:17:11.451+0100	WARN	[cfgwarn]	management/manager.go:126	BETA: Central management is enabled
> 2019-11-14T16:17:11.451+0100	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
> 2019-11-14T16:17:11.451+0100	INFO	[centralmgmt]	management/manager.go:127	Starting central management service
> 2019-11-14T16:17:11.452+0100	INFO	[centralmgmt.event_reporter]	api/event_reporter.go:52	Starting event reporter service
> 2019-11-14T16:17:11.452+0100	INFO	[centralmgmt]	management/manager.go:328	Updating state to 'STARTING'
> 2019-11-14T16:17:11.453+0100	INFO	cfgfile/reload.go:171	Config reloader started
> 2019-11-14T16:17:11.455+0100	INFO	cfgfile/reload.go:226	Loading of config files completed.
> 2019-11-14T16:17:11.842+0100	INFO	[centralmgmt]	management/manager.go:328	Updating state to 'IN_PROGRESS'
> 2019-11-14T16:17:11.842+0100	INFO	[centralmgmt]	management/manager.go:279	Applying settings for output
> 2019-11-14T16:17:11.842+0100	INFO	[centralmgmt]	management/manager.go:279	Applying settings for metricbeat.modules
> 2019-11-14T16:17:11.842+0100	INFO	[centralmgmt]	management/manager.go:328	Updating state to 'RUNNING'
> 2019-11-14T16:17:33.193+0100	INFO	cfgfile/reload.go:229	Dynamic config reloader stopped
> 2019-11-14T16:17:33.193+0100	INFO	[reload]	cfgfile/list.go:118	Stopping 3 runners ...
> 2019-11-14T16:17:33.194+0100	INFO	[centralmgmt]	management/manager.go:141	Stopping central management service
> 2019-11-14T16:17:33.194+0100	INFO	[centralmgmt]	management/manager.go:328	Updating state to 'STOPPED'
> 2019-11-14T16:17:33.194+0100	INFO	[centralmgmt.event_reporter]	api/event_reporter.go:57	Stopping event reporter service
> 2019-11-14T16:17:33.209+0100	INFO	[monitoring]	log/log.go:153	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":147,"time":{"ms":147}},"total":{"ticks":649,"time":{"ms":649},"value":649},"user":{"ticks":502,"time":{"ms":502}}},"info":{"ephemeral_id":"de5d422a-36d0-4d13-b09c-7424e2b7b337","uptime":{"ms":21808}},"memstats":{"gc_next":30466960,"memory_alloc":16180944,"memory_total":823178392,"rss":85471232},"runtime":{"goroutines":15}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"type":""},"pipeline":{"clients":0,"events":{"active":94,"filtered":1,"published":94,"total":95}}},"metricbeat":{"system":{"cpu":{"events":3,"success":3},"filesystem":{"events":5,"success":5},"fsstat":{"events":1,"success":1},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":45,"success":45},"process":{"events":28,"success":28},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3},"uptime":{"events":1,"success":1}}},"system":{"cpu":{"cores":4},"load":{"1":2.0684,"15":2.2085,"5":2.1606,"norm":{"1":0.5171,"15":0.5521,"5":0.5402}}}}}}
> 2019-11-14T16:17:33.209+0100	INFO	[monitoring]	log/log.go:154	Uptime: 21.809316452s

Well I found the issue.

First of all I was using a newer version of metricbeat (7.4.2) than elasticsearch (7.4.1).
I downgraded the metricbeat version and tried to enroll the beat via central management which is overwriting my configuration in a wrong way. MIght be a bug because its still in beta.

Anyway I setup metricbeat via my metricbeat.yml and now it works as it should.
So:

  1. Check if ELK Version is equal to the beat version
  2. I used the metricbeat.yml and not the central management to enroll

@MiTschMR thanks for your help

1 Like