Missing fields from Metricbeat AWS Module

[chicoco (Amr)]

Hello,
I am new to Metricbeat, I want to monitor my AWS services, The overview dashboard is missing a lot of information, I started with EC2 CPU Utilization I found aws.ec2.cpu.total.pct field is missing although I tried with aws-cli and I getting the CPU metrics. not sure for other dashboards what is the problem.

# Module: aws
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/8.4/metricbeat-module-aws.html

- module: aws
  period: 1m
  metricsets:
    - elb
    - kinesis
    - natgateway
    - rds
    - transitgateway
    - usage
    - vpn
- module: aws
  period: 5m
  metricsets:
    - cloudwatch
  latency: 3m
  metrics:
    - namespace: AWS/EC2
      name: ["CPUUtilization", "DiskWriteOps"]
      tags.resource_type_filter: ec2:instance
      statistic: ["Average", "Maximum"]
- module: aws
  period: 300s
  metricsets:
    - dynamodb
    - ebs
    - ec2
    - lambda
    - rds
    - sns
    - sqs
- module: aws
  period: 24h
  metricsets:
    - billing
  cost_explorer_config:
    group_by_dimension_keys:
      - "AZ"
      - "INSTANCE_TYPE"
      - "SERVICE"
      - "LINKED_ACCOUNT"
    group_by_tag_keys:
      - "aws:createdBy"
- module: aws
  period: 24h
  metricsets:
    - s3_daily_storage
- module: aws
  period: 1m
  latency: 5m
  metricsets:
    - s3_request

[jolt]

I can confirm, I am also seeing these issues in our environment

The EC2 CPU metric data appears to be missing from ECE whilst it is present in AWS. Other EC2 metrics have been successfully ingested however.

The AWS billing integration also appears to be missing fields. We are seeing successful collection of the Estimated fields only.

[Kaiyan_Sheng]

Hello! Actually AWS EC2 CPU metric is under host.cpu.usage now and also under host, you should be able to find host.disk.* and host.network.*.

[Kaiyan_Sheng]

The AWS billing integration also appears to be missing fields. We are seeing successful collection of the Estimated fields only.

Interesting! The billing integration should also return metrics from the cost explorer API. Do you by chance seeing any permission related error message in logs that maybe indicates missing permission for the cost explorer API?

[chicoco (Amr)]

Hi Kaiyan, Thanks for your reply, but in this case I will only monitor the host, I would like to monitor all EC2 instances in my account.

[Kaiyan_Sheng]

Please let us know if there are enhancement you'd like to see for the ec2 integration!

[jolt]

Appologies for the delay, it looks like it is related to the ListAccounts permission. We are working within a child subscription so dont have access to that permission despite it being in the iam policy.