Good day,
I'm currently trying to collect AWS metrics with the Metricbeat AWS module. And although the status of metricbeat looks fine, elasticsearch doesn't seem to receive any data.
my metricbeat.yml:
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
processors:
- add_cloud_metadata: ~
- add_docker_metadata: ~
metricbeat.modules:
- module: aws
period: 300s
metricsets:
- ec2
access_key_id: '<access-key-id>'
secret_access_key: '<secret-access-key>'
- module: aws
period: 300s
metricsets:
- cloudwatch
metrics:
- namespace: AWS/EC2
resource_type: ec2:instance
credential_profile_name: <my-profile-name>
access_key_id: '<access-key-id>'
secret_access_key: '<secret-access-key>'
- module: aws
period: 24h
metricsets:
- billing
access_key_id: '<access-key-id>'
secret_access_key: '<secret-access-key>'
cost_explorer_config:
group_by_dimension_keys:
- "AZ"
- "INSTANCE_TYPE"
- "SERVICE"
group_by_tag_keys:
- "aws:createdBy"
output.elasticsearch:
hosts: ["https://localhost:9200"]
username: elastic
password: <my-elastic-pw>
ssl.certificate_authorities: ["<path-to-ca.crt>"]
setup.kibana:
host: "http://localhost:5601"
The output from sudo metricbeat setup -e d "*":
2021-01-05T12:48:24.063+0100 INFO instance/beat.go:645 Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
2021-01-05T12:48:24.063+0100 INFO instance/beat.go:653 Beat ID: 4d5b765b-26cb-4050-bed8-416aa5f320e2
2021-01-05T12:48:24.064+0100 INFO [beat] instance/beat.go:981 Beat info {"system_info": {"beat": {"path": {"config": "/etc/metricbeat", "data": "/var/lib/metricbeat", "home": "/usr/share/metricbeat", "logs": "/var/log/metricbeat"}, "type": "metricbeat", "uuid": "4d5b765b-26cb-4050-bed8-416aa5f320e2"}}}
2021-01-05T12:48:24.064+0100 INFO [beat] instance/beat.go:990 Build info {"system_info": {"build": {"commit": "1428d58cf2ed945441fb2ed03961cafa9e4ad3eb", "libbeat": "7.10.0", "time": "2020-11-09T20:08:47.000Z", "version": "7.10.0"}}}
2021-01-05T12:48:24.064+0100 INFO [beat] instance/beat.go:993 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.14.7"}}}
2021-01-05T12:48:24.065+0100 INFO [beat] instance/beat.go:997 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-11-23T13:52:21+01:00","containerized":false,"name":"<host>","ip":["<ip>","::<more ip>","<ip>","<IPv6>"],"kernel_version":"5.4.0-1029-aws","mac":["<mac>"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"18.04.5 LTS (Bionic Beaver)","major":18,"minor":4,"patch":5,"codename":"bionic"},"timezone":"CET","timezone_offset_sec":3600,"id":"ec2cdb77e116a0d46764c463497dcc2f"}}}
2021-01-05T12:48:24.065+0100 INFO [beat] instance/beat.go:1026 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/etc/metricbeat", "exe": "/usr/share/metricbeat/bin/metricbeat", "name": "metricbeat", "pid": 12929, "ppid": 12928, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2021-01-05T12:48:23.160+0100"}}}
2021-01-05T12:48:24.066+0100 INFO instance/beat.go:299 Setup Beat: metricbeat; Version: 7.10.0
2021-01-05T12:48:24.066+0100 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'metricbeat-7.10.0' as ILM is enabled.
2021-01-05T12:48:24.066+0100 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:93 add_cloud_metadata: hosting provider type detected as aws, metadata={"account":{"id":"<account>"},"availability_zone":"eu-central-1a","image":{"id":"ami-<ami>"},"instance":{"id":"i-<instance>"},"machine":{"type":<type>"},"provider":"aws","region":"<region>"}
2021-01-05T12:48:24.066+0100 INFO eslegclient/connection.go:99 elasticsearch url: https://localhost:9200
2021-01-05T12:48:24.067+0100 INFO [publisher] pipeline/module.go:113 Beat name: <....>
2021-01-05T12:48:24.081+0100 INFO eslegclient/connection.go:99 elasticsearch url: https://localhost:9200
2021-01-05T12:48:24.170+0100 INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.0
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.
2021-01-05T12:48:24.301+0100 INFO [index-management] idxmgmt/std.go:261 Auto ILM enable success.
2021-01-05T12:48:24.309+0100 INFO [index-management.ilm] ilm/std.go:139 do not generate ilm policy: exists=true, overwrite=false
2021-01-05T12:48:24.309+0100 INFO [index-management] idxmgmt/std.go:274 ILM policy successfully loaded.
2021-01-05T12:48:24.309+0100 INFO [index-management] idxmgmt/std.go:407 Set setup.template.name to '{metricbeat-7.10.0 {now/d}-000001}' as ILM is enabled.
2021-01-05T12:48:24.309+0100 INFO [index-management] idxmgmt/std.go:412 Set setup.template.pattern to 'metricbeat-7.10.0-*' as ILM is enabled.
2021-01-05T12:48:24.309+0100 INFO [index-management] idxmgmt/std.go:446 Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.10.0 {now/d}-000001} as ILM is enabled.
2021-01-05T12:48:24.309+0100 INFO [index-management] idxmgmt/std.go:450 Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2021-01-05T12:48:24.319+0100 INFO template/load.go:183 Existing template will be overwritten, as overwrite is enabled.
2021-01-05T12:48:24.600+0100 INFO template/load.go:117 Try loading template metricbeat-7.10.0 to Elasticsearch
2021-01-05T12:48:25.264+0100 INFO template/load.go:109 template with name 'metricbeat-7.10.0' loaded.
2021-01-05T12:48:25.264+0100 INFO [index-management] idxmgmt/std.go:298 Loaded index template.
2021-01-05T12:48:25.272+0100 INFO [index-management] idxmgmt/std.go:309 Write alias successfully generated.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
2021-01-05T12:48:25.273+0100 INFO kibana/client.go:119 Kibana url: http://localhost:5601
2021-01-05T12:48:25.708+0100 INFO kibana/client.go:119 Kibana url: http://localhost:5601
2021-01-05T12:50:06.628+0100 INFO instance/beat.go:815 Kibana dashboards successfully loaded.
Loaded dashboards
The output from sudo service metricbeat status:
● metricbeat.service - Metricbeat is a lightweight shipper for metrics.
Loaded: loaded (/lib/systemd/system/metricbeat.service; disabled; vendor preset: enabled)
Active: active (running) since Tue 2021-01-05 12:50:37 CET; 3s ago
Docs: https://www.elastic.co/products/beats/metricbeat
Main PID: 13015 (metricbeat)
Tasks: 8 (limit: 4915)
CGroup: /system.slice/metricbeat.service
└─13015 /usr/share/metricbeat/bin/metricbeat --environment systemd -c /etc/metricbeat/metricbeat.yml --path.home /usr/share/metricbeat --path.config /etc/metricbeat --path.data /var/lib/metricbeat --path.logs /var/log/metricbeat
Jan 05 12:50:37 <host> metricbeat[13015]: 2021-01-05T12:50:37.177+0100 INFO [beat] instance/beat.go:997 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-11-23T13:52:21+01:00","containerized":false,"n
Jan 05 12:50:37 <host> metricbeat[13015]: 2021-01-05T12:50:37.178+0100 INFO [beat] instance/beat.go:1026 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read
Jan 05 12:50:37 <host> metricbeat[13015]: 2021-01-05T12:50:37.178+0100 INFO instance/beat.go:299 Setup Beat: metricbeat; Version: 7.10.0
Jan 05 12:50:37 <host> metricbeat[13015]: 2021-01-05T12:50:37.178+0100 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'metricbeat-7.10.0' as ILM is enabled.
Jan 05 12:50:37 <host> metricbeat[13015]: 2021-01-05T12:50:37.179+0100 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:93 add_cloud_metadata: hosting provider type detected as aws, metadata={"account":{"id":"<id>
Jan 05 12:50:37 <host> metricbeat[13015]: 2021-01-05T12:50:37.179+0100 INFO eslegclient/connection.go:99 elasticsearch url: https://localhost:9200
Jan 05 12:50:37 <host> metricbeat[13015]: 2021-01-05T12:50:37.179+0100 INFO [publisher] pipeline/module.go:113 Beat name: <....>
Jan 05 12:50:37 <host> metricbeat[13015]: 2021-01-05T12:50:37.198+0100 WARN [aws.ec2] aws/aws.go:99 extra charges on AWS API requests will be generated by this metricset
Jan 05 12:50:38 <host> metricbeat[13015]: 2021-01-05T12:50:38.414+0100 WARN [aws.cloudwatch] aws/aws.go:99 extra charges on AWS API requests will be generated by this metricset
Jan 05 12:50:39 <host> metricbeat[13015]: 2021-01-05T12:50:39.572+0100 WARN [aws.billing] aws/aws.go:99 extra charges on AWS API requests will be generated by this metricset
But Kibana tells me that: No data has been received from this module yet
And there is no matching index created.
I running out of ideas here and would appreciate any help.
Looking at your config, the ec2 metricset part and cloudwatch metricset part are basically doing the same thing so you are actually trying to collect ec2 metrics twice with this config. Also in your config, you have 