Kibana version : 7.10.0
Elasticsearch version : 7.10.0
Metricbeat version : 7.10.1
Good day,
I'm trying to collect monitoring metrics from AWS with metricbeat. In particular, I am interested in the metrics of the 5 EC2 instances on AWS and the billing informations.
I'm running metricbeat on one of the EC2 machines in the AWS cluster.
my metricbeat.yml:
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
processors:
- add_cloud_metadata: ~
- add_docker_metadata: ~
metricbeat.modules:
- module: aws
period: 300s
metricsets:
- ec2
access_key_id: '<my-access-key-id>'
secret_access_key: '<my-secret-access-key>'
- module: aws
period: 24h
metricsets:
- billing
access_key_id: '<my-access-key-id>'
secret_access_key: '<secret-access-key>'
cost_explorer_config:
group_by_dimension_keys:
- "AZ"
- "INSTANCE_TYPE"
- "SERVICE"
group_by_tag_keys:
- "aws:createdBy"
output.elasticsearch:
hosts: ["https://localhost:9200"]
username: elastic
password: <my-pw>
ssl.certificate_authorities: ["<path-to-my-ca.crt>"]
setup.kibana:
host: "http://localhost:5601"
my metricbeat setup logs from sudo metricbeat setup -e d "*":
2020-12-27T14:24:47.577+0100 INFO instance/beat.go:645 Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
2020-12-27T14:24:47.577+0100 INFO instance/beat.go:653 Beat ID: 4d5b765b-26cb-4050-bed8-416aa5f320e2
2020-12-27T14:24:47.578+0100 INFO [beat] instance/beat.go:981 Beat info {"system_info": {"beat": {"path": {"config": "/etc/metricbeat", "data": "/var/lib/metricbeat", "home": "/usr/share/metricbeat", "logs": "/var/log/metricbeat"}, "type": "metricbeat", "uuid": "4d5b765b-26cb-4050-bed8-416aa5f320e2"}}}
2020-12-27T14:24:47.578+0100 INFO [beat] instance/beat.go:990 Build info {"system_info": {"build": {"commit": "1428d58cf2ed945441fb2ed03961cafa9e4ad3eb", "libbeat": "7.10.0", "time": "2020-11-09T20:08:47.000Z", "version": "7.10.0"}}}
2020-12-27T14:24:47.578+0100 INFO [beat] instance/beat.go:993 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.14.7"}}}
2020-12-27T14:24:47.579+0100 INFO [beat] instance/beat.go:997 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-11-23T13:52:21+01:00","containerized":false,"name":"abby.yatta.de","ip":["127.0.0.1/8","::1/128","172.31.30.230/20","fe80::5f:f7ff:fe57:bc5a/64"],"kernel_version":"5.4.0-1029-aws","mac":["02:5f:f7:57:bc:5a"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"18.04.5 LTS (Bionic Beaver)","major":18,"minor":4,"patch":5,"codename":"bionic"},"timezone":"CET","timezone_offset_sec":3600,"id":"ec2cdb77e116a0d46764c463497dcc2f"}}}
2020-12-27T14:24:47.579+0100 INFO [beat] instance/beat.go:1026 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/etc/metricbeat", "exe": "/usr/share/metricbeat/bin/metricbeat", "name": "metricbeat", "pid": 25899, "ppid": 25898, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2020-12-27T14:24:46.680+0100"}}}
2020-12-27T14:24:47.579+0100 INFO instance/beat.go:299 Setup Beat: metricbeat; Version: 7.10.0
2020-12-27T14:24:47.579+0100 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'metricbeat-7.10.0' as ILM is enabled.
2020-12-27T14:24:47.580+0100 INFO eslegclient/connection.go:99 elasticsearch url: https://localhost:9200
2020-12-27T14:24:47.580+0100 INFO [publisher] pipeline/module.go:113 Beat name: abby.yatta.de
2020-12-27T14:24:47.581+0100 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:93 add_cloud_metadata: hosting provider type detected as aws, metadata={"account":{"id":"039697085671"},"availability_zone":"eu-central-1a","image":{"id":"ami-067bf89ce72d3b286"},"instance":{"id":"i-061bb4789d34bd527"},"machine":{"type":"r4.xlarge"},"provider":"aws","region":"eu-central-1"}
2020-12-27T14:24:47.597+0100 INFO eslegclient/connection.go:99 elasticsearch url: https://localhost:9200
2020-12-27T14:24:47.677+0100 INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.0
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.
2020-12-27T14:24:47.764+0100 INFO [index-management] idxmgmt/std.go:261 Auto ILM enable success.
2020-12-27T14:24:47.773+0100 INFO [index-management.ilm] ilm/std.go:139 do not generate ilm policy: exists=true, overwrite=false
2020-12-27T14:24:47.773+0100 INFO [index-management] idxmgmt/std.go:274 ILM policy successfully loaded.
2020-12-27T14:24:47.773+0100 INFO [index-management] idxmgmt/std.go:407 Set setup.template.name to '{metricbeat-7.10.0 {now/d}-000001}' as ILM is enabled.
2020-12-27T14:24:47.773+0100 INFO [index-management] idxmgmt/std.go:412 Set setup.template.pattern to 'metricbeat-7.10.0-*' as ILM is enabled.
2020-12-27T14:24:47.773+0100 INFO [index-management] idxmgmt/std.go:446 Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.10.0 {now/d}-000001} as ILM is enabled.
2020-12-27T14:24:47.773+0100 INFO [index-management] idxmgmt/std.go:450 Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-12-27T14:24:47.785+0100 INFO template/load.go:183 Existing template will be overwritten, as overwrite is enabled.
2020-12-27T14:24:48.060+0100 INFO template/load.go:117 Try loading template metricbeat-7.10.0 to Elasticsearch
2020-12-27T14:24:48.688+0100 INFO template/load.go:109 template with name 'metricbeat-7.10.0' loaded.
2020-12-27T14:24:48.688+0100 INFO [index-management] idxmgmt/std.go:298 Loaded index template.
2020-12-27T14:24:48.699+0100 INFO [index-management] idxmgmt/std.go:309 Write alias successfully generated.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
2020-12-27T14:24:48.700+0100 INFO kibana/client.go:119 Kibana url: http://localhost:5601
2020-12-27T14:24:49.115+0100 INFO kibana/client.go:119 Kibana url: http://localhost:5601
2020-12-27T14:26:30.924+0100 INFO instance/beat.go:815 Kibana dashboards successfully loaded.
Loaded dashboards
But unfortunately I don't get any data from metricbeat. The index is empty and kibana tells me: No data has been received from this module yet.
I assume that I'm missing the necessary rights for these API calls.
So my question is: what rigths do I need to make these API calls, so I can tell my AWS Admin to configure the necessary rights. 
Thank you in advance for any help.