Can't see consulbeat import any data

Elastic Stack Beats
1

Hi,

I use consul in my project so I was happy when I found the consulbeat.

I tried to us it together with ES 5.3.1.

Unfortunately i can't see that it is doing anything.
No index is created and no error in the log.

cat /var/log/consulbeat/consulbeat
2017-04-24T06:18:00Z INFO Metrics logging every 30s
2017-04-24T06:18:00Z INFO Loading template enabled. Reading template file: /etc/consulbeat/consulbeat.template.json
2017-04-24T06:18:00Z INFO Elasticsearch url: http://elasticsearch.service.consul:9200
2017-04-24T06:18:00Z INFO Activated elasticsearch as output plugin.
2017-04-24T06:18:00Z INFO Publisher name: consulbeat
2017-04-24T06:18:00Z INFO Flush Interval set to: 1s
2017-04-24T06:18:00Z INFO Max Bulk Size set to: 50
2017-04-24T06:18:00Z INFO consulbeat start running.
2017-04-24T06:18:00Z INFO consulbeat is running! Hit CTRL-C to stop it.
2017-04-24T06:18:01Z INFO Event sent
2017-04-24T06:18:02Z INFO Event sent
2017-04-24T06:18:03Z INFO Event sent
2017-04-24T06:18:04Z INFO Event sent

curl elasticsearch:9200/_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open heartbeat-2017.04.24 A5JC9DAeQLSbcdDQDYz9xQ 5 1 44318 0 18.5mb 18.5mb
yellow open metricbeat-2017.04.24 8SgsrGjPQcCEsXvdS42RJA 5 1 1043073 0 776.1mb 776.1mb
yellow open .kibana pRToTA8-QFGcNweqVEhtUA 5 1 148 96 521.8kb 521.8kb
yellow open filebeat-2017.04.24 Js56v6rcTU22z5AzJ5qPKg 5 1 263402 0 107mb 107mb

Are there any special trick to make this beat work?

Br Mathias

2

Can you run consulbeat with debug output enabled?

3

Hi Steffens,

cat /var/log/consulbeat/consulbeat
2017-04-25T17:33:39Z INFO Metrics logging every 30s
2017-04-25T17:33:39Z INFO Home path: [/usr/share/consulbeat] Config path: [/etc/consulbeat] Data path: [/var/lib/consulbeat] Logs path: [/var/log/consulbeat]
2017-04-25T17:33:39Z INFO Setup Beat: consulbeat; Version: 5.0.0
2017-04-25T17:33:39Z DBG Processors:
2017-04-25T17:33:39Z DBG Initializing output plugins
2017-04-25T17:33:39Z INFO Loading template enabled. Reading template file: /etc/consulbeat/consulbeat.template.json
2017-04-25T17:33:39Z INFO Elasticsearch url: http://consul.service.consul:9200
2017-04-25T17:33:39Z INFO Activated elasticsearch as output plugin.
2017-04-25T17:33:39Z DBG Create output worker
2017-04-25T17:33:39Z DBG No output is defined to store the topology. The server fields might not be filled.
2017-04-25T17:33:39Z INFO Publisher name: consulbeat
2017-04-25T17:33:39Z INFO Flush Interval set to: 1s
2017-04-25T17:33:39Z INFO Max Bulk Size set to: 50
2017-04-25T17:33:39Z DBG create bulk processing worker (interval=1s, bulk size=50)
2017-04-25T17:33:39Z INFO consulbeat start running.
2017-04-25T17:33:39Z INFO consulbeat is running! Hit CTRL-C to stop it.
2017-04-25T17:33:40Z DBG No events to publish
2017-04-25T17:33:40Z INFO Event sent
2017-04-25T17:33:41Z DBG No events to publish
2017-04-25T17:33:41Z INFO Event sent
2017-04-25T17:33:42Z DBG No events to publish
2017-04-25T17:33:42Z INFO Event sent
2017-04-25T17:33:43Z DBG No events to publish
2017-04-25T17:33:43Z INFO Event sent
2017-04-25T17:33:44Z DBG No events to publish

I also try to reload consul but there is no events.

Br Mathias

4

I haven't used this beat myself yet, so no idea about potential configuration errors. But to me it looks like it's not fetching any data or filtering out all data (or let's say an empty batch).

The debug message No events to publish is written by the libbeat publisher pipeline right after filtering. Either all events have been filtered out, or consulbeat did try to publish an empty batch of events.