Metricbeat is starting, appears to be running, and I see no errors in the logs.
[root@xxx metricbeat]# metricbeat test config
Config OK
[root@xxx metricbeat]# metricbeat test modules
[root@xxx metricbeat]# metricbeat modules list
Enabled:
aws
Disabled:
[root@xxx metricbeat]# metricbeat test output
Client 0...
logstash: logstash-xxx.com:5044...
connection...
parse host... OK
dns lookup... OK
addresses: x.x.x.x, y.y.y.y, z.z.z.z
dial up... OK
TLS... WARN secure connection disabled
talk to server... OK
Client 1...
logstash: logstash-xxx.com:5044...
connection...
parse host... OK
dns lookup... OK
addresses: x.x.x.x, y.y.y.y, z.z.z.z
dial up... OK
TLS... WARN secure connection disabled
talk to server... OK
Client 2...
logstash: logstash-xxx.com:5044...
connection...
parse host... OK
dns lookup... OK
addresses: x.x.x.x, y.y.y.y, z.z.z.z
dial up... OK
TLS... WARN secure connection disabled
talk to server... OK
Something isn't working though, as I do not see any records in Elasticsearch.
I don't know if any of the instances of the aws module are loading properly from the configuration. I can only see as per metricbeat modules list that the aws module is Enabled. It always says that, though, even if I provide an empty modules.d/aws.yml file.
I don't know what to make of the log output. I certainly don't see a problem there:
2019-12-06T00:56:35.759Z INFO [beat] instance/beat.go:948 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/metricbeat", "exe": "/usr/share/metricbeat/metricbeat", "name": "metricbeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2019-12-06T00:56:35.460Z"}}}
2019-12-06T00:56:35.759Z INFO instance/beat.go:292 Setup Beat: metricbeat; Version: 7.4.2
2019-12-06T00:56:35.759Z INFO [publisher] pipeline/module.go:97 Beat name: metricbeat
2019-12-06T00:56:35.760Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2019-12-06T00:56:35.760Z INFO instance/beat.go:422 metricbeat start running.
2019-12-06T00:56:35.760Z INFO cfgfile/reload.go:171 Config reloader started
2019-12-06T00:57:05.762Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":50,"time":{"ms":55}},"total":{"ticks":260,"time":{"ms":270},"value":0},"user":{"ticks":210,"time":{"ms":215}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":5},"info":{"ephemeral_id":"x-x-x-x-x","uptime":{"ms":30045}},"memstats":{"gc_next":11475504,"memory_alloc":8201760,"memory_total":14789104,"rss":69169152},"runtime":{"goroutines":20}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":8},"load":{"1":0.32,"15":0.06,"5":0.14,"norm":{"1":0.04,"15":0.0075,"5":0.0175}}}}}}
The metricbeat process runs like this:
metricbeat -c /usr/share/metricbeat/metricbeat.yml -e -d "*"
My metricbeat.yml looks like this:
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
name: metricbeat
fields:
environment: xxx
processors:
- add_cloud_metadata:
output.elasticsearch:
enabled: false
output.logstash:
enabled: true
hosts: ["logstash-xxx.com:5044","logstash-xxx.com:5044","logstash-xxx.com:5044"]
My modules.d/aws.yml looks like this:
metricbeat.modules:
- module: aws
fields:
environment: "yyy"
period: 300s
metricsets:
- cloudwatch
access_key_id: "xxx"
secret_access_key: "xxx"
session_token: '${AWS_SESSION_TOKEN:""}'
default_region: "eu-west-1"
cloudwatch_metrics:
- namespace: AWS/EC2
How do I begin to troubleshoot this? I have no idea what data metricbeat is attempting to collect, successfully or unsuccessfully. I don't have a clue what its doing.
