Heartbeat is not sending anything to Elasticsearch

Elastic Stack Beats
1

Hi,

I think I am missing something.
I have started heartbeat. I see that dashboards are created in Kibana but there is no data from heartbeat.

What is missing?

heartbeat.monitors:

output.elasticsearch:
hosts: ["elasticsearch.service.consul:9200"]

logging.level: debug
logging.selectors: ["*"]
logging.metrics.enabled: true
logging.to_files: true
logging.files:
path: /var/log/heartbeat
name: heartbeat

Are there any more configuration I need to do?

This is the log:

2017-04-21T05:42:00Z INFO Kibana dashboards successfully loaded.
2017-04-21T05:42:00Z INFO heartbeat start running.
2017-04-21T05:42:00Z INFO heartbeat is running! Hit CTRL-C to stop it.
2017-04-21T05:42:00Z DBG Start scheduler.
2017-04-21T05:42:29Z INFO Non-zero metrics in the last 30s: libbeat.es.publish.read_bytes=4432 libbeat.es.publish.write_bytes=22521
2017-04-21T05:42:59Z INFO No non-zero metrics in the last 30s
2017-04-21T05:43:29Z INFO No non-zero metrics in the last 30s
2017-04-21T05:43:59Z INFO No non-zero metrics in the last 30s
2017-04-21T05:44:29Z INFO No non-zero metrics in the last 30s
2017-04-21T05:44:59Z INFO No non-zero metrics in the last 30s
2017-04-21T05:45:29Z INFO No non-zero metrics in the last 30s
2017-04-21T05:45:59Z INFO No non-zero metrics in the last 30s
2017-04-21T05:46:29Z INFO No non-zero metrics in the last 30s
2017-04-21T05:46:59Z INFO No non-zero metrics in the last 30s
2017-04-21T05:47:29Z INFO No non-zero metrics in the last 30s
2017-04-21T05:47:59Z INFO No non-zero metrics in the last 30s
2017-04-21T05:48:29Z INFO No non-zero metrics in the last 30s
2017-04-21T05:48:59Z INFO No non-zero metrics in the last 30s
2017-04-21T05:49:29Z INFO No non-zero metrics in the last 30

curl elasticsearch:9200/_cat/indices?v

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open .kibana NjBHLO5kQk6Za0hNEJFmHA 5 1 110 71 376.1kb 376.1kb
yellow open metricbeat-2017.04.21 BNilHxvPRJGKdJXL-KmyWg 5 1 208947 0 141mb 141mb
yellow open filebeat-2017.04.21 2xXKrGslRWGs2_4Jm6IULA 5 1 16808 0 16.7mb 16.7mb

Br Mathias

2

Is this the complete debug log? Enabling all selectors, heartbeat should log jobs being created and log a message whenever a job is run + log the actual event being published.

I can't tell anything wrong from config file.

Setting debug selectors on command line via -d "*,config" will print the configuration in JSON format for better inspection.

3

Hi Steffens,

Here comes -d "*,config" output

heartbeat.sh -d ",config"
heartbeat2017/04/21 12:50:58.956008 config.go:275: DBG CLI setting overwrites (-E flag):
{
"path": {
"config": "/etc/heartbeat",
"data": "/var/lib/heartbeat",
"home": "/usr/share/heartbeat",
"logs": "/var/log/heartbeat"
}
}
heartbeat2017/04/21 12:50:58.957470 config.go:275: DBG load config file '/etc/heartbeat/heartbeat.yml' =>
{
"dashboards": {
"enabled": true,
"index": "heartbeat-"
},
"heartbeat": {
"monitors": [
{
"schedule": "@every 5s",
"timeout": "16s",
"type": "http",
"urls": [
"xxxxx",
"xxxxx"
]
}
]
},
"logging": {
"files": {
"keepfiles": 2,
"name": "heartbeat",
"path": "/var/log/heartbeat",
"rotateeverybytes": 10485760
},
"level": "debug",
"metrics": {
"enabled": true
},
"selectors": [
""
],
"to_files": true
},
"name": "heartbeat",
"output": {
"elasticsearch": {
"hosts": [
"xxxxx"
]
}
}
}
heartbeat2017/04/21 12:50:58.957623 config.go:275: DBG Complete configuration loaded:
{
"dashboards": {
"enabled": true,
"index": "heartbeat-"
},
"heartbeat": {
"monitors": [
{
"schedule": "@every 5s",
"timeout": "16s",
"type": "http",
"urls": [
"xxxxx",
"xxxxx"
]
}
]
},
"logging": {
"files": {
"keepfiles": 2,
"name": "heartbeat",
"path": "/var/log/heartbeat",
"rotateeverybytes": 10485760
},
"level": "debug",
"metrics": {
"enabled": true
},
"selectors": [
"*"
],
"to_files": true
},
"name": "heartbeat",
"output": {
"elasticsearch": {
"hosts": [
"xxxxx"
]
}
},
"path": {
"config": "/etc/heartbeat",
"data": "/var/lib/heartbeat",
"home": "/usr/share/heartbeat",
"logs": "/var/log/heartbeat"
}
}

From this I get the following log:

sudo cat /var/log/heartbeat/heartbeat
2017-04-21T11:36:02Z INFO Metrics logging every 30s
2017-04-21T11:36:02Z INFO Home path: [/usr/share/heartbeat] Config path: [/etc/heartbeat] Data path: [/var/lib/heartbeat] Logs path: [/var/log/heartbeat]
2017-04-21T11:36:02Z INFO Setup Beat: heartbeat; Version: 5.3.1
2017-04-21T11:36:02Z DBG Processors:
2017-04-21T11:36:02Z DBG Initializing output plugins
2017-04-21T11:36:02Z DBG Configure output plugin 'elasticsearch' with:
{
"hosts": [
"xxxxx"
]
}
2017-04-21T11:36:02Z INFO Loading template enabled. Reading template file: /etc/heartbeat/heartbeat.template.json
2017-04-21T11:36:02Z INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /etc/heartbeat/heartbeat.template-es2x.json
2017-04-21T11:36:02Z INFO Elasticsearch url: http://elasticsearch.service.consul:9200
2017-04-21T11:36:02Z INFO Activated elasticsearch as output plugin.
2017-04-21T11:36:02Z DBG Create output worker
2017-04-21T11:36:02Z DBG No output is defined to store the topology. The server fields might not be filled.
2017-04-21T11:36:02Z INFO Publisher name: heartbeat
2017-04-21T11:36:02Z INFO Flush Interval set to: 1s
2017-04-21T11:36:02Z INFO Max Bulk Size set to: 50
2017-04-21T11:36:02Z DBG create bulk processing worker (interval=1s, bulk size=50)
2017-04-21T11:36:02Z WARN Beta: Heartbeat is beta software
2017-04-21T11:36:02Z INFO Select (active) monitor http
2017-04-21T11:36:02Z ERR failed to load monitor tasks: missing port in address when initializing monitor http(0)
2017-04-21T11:36:02Z INFO Elasticsearch url: http://elasticsearch.service.consul:9200
2017-04-21T11:36:02Z DBG ES Ping(url=http://elasticsearch.service.consul:9200, timeout=1m30s)
2017-04-21T11:36:02Z DBG Ping request failed with: Get http://elasticsearch.service.consul:9200: dial tcp 10.68.32.181:9200: getsockopt: connection refused
2017-04-21T11:36:02Z ERR Error connecting to Elasticsearch: http://elasticsearch.service.consul:9200
2017-04-21T11:36:02Z CRIT Exiting: Error creating ES client: Couldn't connect to any of the configured > Elasticsearch hosts

It is complaining about port number:
ERR failed to load monitor tasks: missing port in address when initializing monitor http(0)

Br Mathias

4

Problem found!

I had missed http:// in the following line:
urls: ["http://consul.service.consul:8500","elasticsearch.service.consul:9200"]

It should be:
urls: ["http://consul.service.consul:8500","http://elasticsearch.service.consul:9200"]

Br Mathias

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.