Metricbeat AWS module

Kibana version : 7.10.0

Elasticsearch version : 7.10.0

Metricbeat version : 7.10.1

Good day,

I'm trying to collect monitoring metrics from AWS with metricbeat. In particular, I am interested in the metrics of the 5 EC2 instances on AWS and the billing informations.

I'm running metricbeat on one of the EC2 machines in the AWS cluster.

my metricbeat.yml:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

processors:
  - add_cloud_metadata: ~
  - add_docker_metadata: ~

metricbeat.modules:
- module: aws
  period: 300s
  metricsets:
    - ec2
  access_key_id: '<my-access-key-id>'
  secret_access_key: '<my-secret-access-key>'

- module: aws
  period: 24h
  metricsets:
    - billing
  access_key_id: '<my-access-key-id>'
  secret_access_key: '<secret-access-key>'
  cost_explorer_config:
    group_by_dimension_keys:
      - "AZ"
      - "INSTANCE_TYPE"
      - "SERVICE"
    group_by_tag_keys:
      - "aws:createdBy"

output.elasticsearch:
  hosts: ["https://localhost:9200"]
  username: elastic
  password: <my-pw>
  ssl.certificate_authorities: ["<path-to-my-ca.crt>"]

setup.kibana:
    host: "http://localhost:5601"

my metricbeat setup logs from sudo metricbeat setup -e d "*":

2020-12-27T14:24:47.577+0100    INFO    instance/beat.go:645    Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
2020-12-27T14:24:47.577+0100    INFO    instance/beat.go:653    Beat ID: 4d5b765b-26cb-4050-bed8-416aa5f320e2
2020-12-27T14:24:47.578+0100    INFO    [beat]  instance/beat.go:981    Beat info       {"system_info": {"beat": {"path": {"config": "/etc/metricbeat", "data": "/var/lib/metricbeat", "home": "/usr/share/metricbeat", "logs": "/var/log/metricbeat"}, "type": "metricbeat", "uuid": "4d5b765b-26cb-4050-bed8-416aa5f320e2"}}}
2020-12-27T14:24:47.578+0100    INFO    [beat]  instance/beat.go:990    Build info      {"system_info": {"build": {"commit": "1428d58cf2ed945441fb2ed03961cafa9e4ad3eb", "libbeat": "7.10.0", "time": "2020-11-09T20:08:47.000Z", "version": "7.10.0"}}}
2020-12-27T14:24:47.578+0100    INFO    [beat]  instance/beat.go:993    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.14.7"}}}
2020-12-27T14:24:47.579+0100    INFO    [beat]  instance/beat.go:997    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-11-23T13:52:21+01:00","containerized":false,"name":"abby.yatta.de","ip":["127.0.0.1/8","::1/128","172.31.30.230/20","fe80::5f:f7ff:fe57:bc5a/64"],"kernel_version":"5.4.0-1029-aws","mac":["02:5f:f7:57:bc:5a"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"18.04.5 LTS (Bionic Beaver)","major":18,"minor":4,"patch":5,"codename":"bionic"},"timezone":"CET","timezone_offset_sec":3600,"id":"ec2cdb77e116a0d46764c463497dcc2f"}}}
2020-12-27T14:24:47.579+0100    INFO    [beat]  instance/beat.go:1026   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/etc/metricbeat", "exe": "/usr/share/metricbeat/bin/metricbeat", "name": "metricbeat", "pid": 25899, "ppid": 25898, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2020-12-27T14:24:46.680+0100"}}}
2020-12-27T14:24:47.579+0100    INFO    instance/beat.go:299    Setup Beat: metricbeat; Version: 7.10.0
2020-12-27T14:24:47.579+0100    INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'metricbeat-7.10.0' as ILM is enabled.
2020-12-27T14:24:47.580+0100    INFO    eslegclient/connection.go:99    elasticsearch url: https://localhost:9200
2020-12-27T14:24:47.580+0100    INFO    [publisher]     pipeline/module.go:113  Beat name: abby.yatta.de
2020-12-27T14:24:47.581+0100    INFO    [add_cloud_metadata]    add_cloud_metadata/add_cloud_metadata.go:93     add_cloud_metadata: hosting provider type detected as aws, metadata={"account":{"id":"039697085671"},"availability_zone":"eu-central-1a","image":{"id":"ami-067bf89ce72d3b286"},"instance":{"id":"i-061bb4789d34bd527"},"machine":{"type":"r4.xlarge"},"provider":"aws","region":"eu-central-1"}
2020-12-27T14:24:47.597+0100    INFO    eslegclient/connection.go:99    elasticsearch url: https://localhost:9200
2020-12-27T14:24:47.677+0100    INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.10.0
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.
2020-12-27T14:24:47.764+0100    INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2020-12-27T14:24:47.773+0100    INFO    [index-management.ilm]  ilm/std.go:139  do not generate ilm policy: exists=true, overwrite=false
2020-12-27T14:24:47.773+0100    INFO    [index-management]      idxmgmt/std.go:274      ILM policy successfully loaded.
2020-12-27T14:24:47.773+0100    INFO    [index-management]      idxmgmt/std.go:407      Set setup.template.name to '{metricbeat-7.10.0 {now/d}-000001}' as ILM is enabled.
2020-12-27T14:24:47.773+0100    INFO    [index-management]      idxmgmt/std.go:412      Set setup.template.pattern to 'metricbeat-7.10.0-*' as ILM is enabled.
2020-12-27T14:24:47.773+0100    INFO    [index-management]      idxmgmt/std.go:446      Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.10.0 {now/d}-000001} as ILM is enabled.
2020-12-27T14:24:47.773+0100    INFO    [index-management]      idxmgmt/std.go:450      Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-12-27T14:24:47.785+0100    INFO    template/load.go:183    Existing template will be overwritten, as overwrite is enabled.
2020-12-27T14:24:48.060+0100    INFO    template/load.go:117    Try loading template metricbeat-7.10.0 to Elasticsearch
2020-12-27T14:24:48.688+0100    INFO    template/load.go:109    template with name 'metricbeat-7.10.0' loaded.
2020-12-27T14:24:48.688+0100    INFO    [index-management]      idxmgmt/std.go:298      Loaded index template.
2020-12-27T14:24:48.699+0100    INFO    [index-management]      idxmgmt/std.go:309      Write alias successfully generated.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
2020-12-27T14:24:48.700+0100    INFO    kibana/client.go:119    Kibana url: http://localhost:5601
2020-12-27T14:24:49.115+0100    INFO    kibana/client.go:119    Kibana url: http://localhost:5601
2020-12-27T14:26:30.924+0100    INFO    instance/beat.go:815    Kibana dashboards successfully loaded.
Loaded dashboards

But unfortunately I don't get any data from metricbeat. The index is empty and kibana tells me: No data has been received from this module yet.

I assume that I'm missing the necessary rights for these API calls.

So my question is: what rigths do I need to make these API calls, so I can tell my AWS Admin to configure the necessary rights. :slight_smile:

Thank you in advance for any help.

According to these intstructions: https://docs.logz.io/shipping/metrics-sources/aws-classic-elb.html

An IAM user with follwowing permissions is needed:

  • cloudwatch:GetMetricData
  • cloudwatch:ListMetrics
  • ec2:DescribeInstances
  • ec2:DescribeRegions
  • iam:ListAccountAliases
  • sts:GetCallerIdentity

I will try to get the necessary permissions and check again with metricbeat.

Please see https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-aws-ec2.html#_aws_permissions_4 and https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-aws-billing.html#_aws_permissions.

Tank you for your help @shaunak.