# Metricbeat AWS Module - billing metricset

**URL:** <https://discuss.elastic.co/t/metricbeat-aws-module-billing-metricset/263223>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [February 4, 2021, 9:54am UTC](https://discuss.elastic.co/t/metricbeat-aws-module-billing-metricset/263223 "2021-02-04T09:54:48Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jurilz](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@Jurilz](https://discuss.elastic.co/u/Jurilz)\
**Post date:** [February 4, 2021, 9:54am UTC](https://discuss.elastic.co/t/metricbeat-aws-module-billing-metricset/263223/1 "2021-02-04T09:54:49Z")

</div>

Good day,

I'm currently trying to collect billing information from AWS by using the billing metricset of the Metricbeat AWS module, but the received data does not contain any billing information.

my metricbeat.yml:

```auto
metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

setup.kibana:
  host: "localhost:5601"

output.elasticsearch:
  hosts: ["localhost:9200"]
  protocol: "https"
  username: elastic
  password: <pw>
  ssl.certificate_authorities: ["<path-to>/ca.crt"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

```

The aws module ist enabled. My /etc/metricbeat/modules.d/aws.yml:

```auto
- module: aws
  period: 24h
  metricsets:
    - billing
  cost_explorer_config:
    group_by_dimension_keys:
      - "AZ"
      - "INSTANCE_TYPE"
      - "SERVICE"
    group_by_tag_keys:
      - "aws:createdBy"
  access_key_id: '<access-key>'
  secret_access_key: '<secret-access-key>'

```

I expected to receive data that contains fields like:

```auto
{
    "aws": {
        "billing": {
            "AmortizedCost": {...},
            "BlendedCost": {...}},
   "event": {
        "dataset": "aws.billing",..
    },
    "metricset": {
        "name": "billing",
}}

```

But instead my I don't receive any billing information. My event.dataset and metricsets look like this:

```auto
  "event": {
        "dataset": "system.process",
        "dataset": "system.network",
        "dataset": "system.memory",
        "dataset": "system.cpu",
        "dataset": "system.load",
    },
  "metricset": {
        "name": "system.process",
        "name": "system.network",
        "name": "system.memory",
        "name": "system.cpu",
        "name": "system.load",
    },

```

My AWS Admin assures me, that I have the necessary rights:

```auto
cloudwatch:GetMetricData
cloudwatch:ListMetrics
tag:getResources
sts:GetCallerIdentity
iam:ListAccountAliases
ce:GetCostAndUsage

```

I receive the same data, when I use the ec2 metricset or the cloudwatch metricset. It doesn't seem to make any difference which metricset I set up.

**Edit** : I also get only data about one instance, although there are three instances deployed on aws.

I am at my wit's end and would appreciate any help.

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [February 8, 2021, 10:25am UTC](https://discuss.elastic.co/t/metricbeat-aws-module-billing-metricset/263223/2 "2021-02-08T10:25:24Z")

</div>

Hi!

Can you please run in debug mode and share any helpful logs?

@Kaiyan_Sheng do you see anything obvious here?

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [February 8, 2021, 2:50pm UTC](https://discuss.elastic.co/t/metricbeat-aws-module-billing-metricset/263223/3 "2021-02-08T14:50:19Z")

</div>

Hello! Could you also run `./metricbeat modules list` to see what are the modules enabled please? If you are only interested in AWS billing, you can disable the system module by running `./metricbeat modules disable system`.

What version of Metricbeat are you running? TIA!

---

<div class="post-metadata">

**Author:** ![Jurilz](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@Jurilz](https://discuss.elastic.co/u/Jurilz)\
**Post date:** [February 8, 2021, 8:17pm UTC](https://discuss.elastic.co/t/metricbeat-aws-module-billing-metricset/263223/4 "2021-02-08T20:17:13Z")

</div>

Thank you for your support.

Metricbeat Version: 7.10.0

I disabled the system module on your advise, so the output from `metricbeat modules list` is:

```auto
Enabled:
aws

```

I have set the loglevel to debug. After the restart of Metricbeat I finally received the billing metricset. 😀

I don't get the cloudwatch AWS/EC2 metricset, which I also request. But this topic is maybe for another thread.

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [February 8, 2021, 10:38pm UTC](https://discuss.elastic.co/t/metricbeat-aws-module-billing-metricset/263223/5 "2021-02-08T22:38:27Z")

</div>

Interesting! If you see metrics in CloudWatch for a specific AWS EC2 instance, it should be able to pick up by the ec2 metricset as well. Here is what the `aws.yml` config should look like for collecting both billing and ec2 metrics:

```auto
- module: aws
  period: 24h
  metricsets:
    - billing
  cost_explorer_config:
    group_by_dimension_keys:
      - "AZ"
      - "INSTANCE_TYPE"
      - "SERVICE"
    group_by_tag_keys:
      - "aws:createdBy"
  access_key_id: '<access-key>'
  secret_access_key: '<secret-access-key>'
- module: aws
  period: 5m
  metricsets:
    - ec2
  access_key_id: '<access-key>'
  secret_access_key: '<secret-access-key>'

```

---

<div class="post-metadata">

**Author:** ![Jurilz](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@Jurilz](https://discuss.elastic.co/u/Jurilz)\
**Post date:** [February 9, 2021, 9:03am UTC](https://discuss.elastic.co/t/metricbeat-aws-module-billing-metricset/263223/6 "2021-02-09T09:03:20Z")

</div>

That would be awesome.

This is my current `aws.yml`:

```auto
- module: aws
  period: 24h
  metricsets:
    - billing
  cost_explorer_config:
    group_by_dimension_keys:
      - "AZ"
      - "INSTANCE_TYPE"
      - "SERVICE"
    group_by_tag_keys:
      - "aws:createdBy"
  access_key_id: '<access_key>'
  secret_access_key: '<ecret_access_key>'
- module: aws
  period: 5m
  metricsets:
    - ec2
  access_key_id: '<access_key>'
  secret_access_key: '<ecret_access_key>'

```

This ist the output from `service metricbeat status`:

```auto
Feb 09 09:54:57 <host> metricbeat[10917]: 2021-02-09T09:54:57.561+0100 INFO cfgfile/reload.go:164 Config reloader started
Feb 09 09:54:57 <host> metricbeat[10917]: 2021-02-09T09:54:57.561+0100 DEBUG [cfgfile] cfgfile/reload.go:194 Scan for new config files
Feb 09 09:54:57 <host> metricbeat[10917]: 2021-02-09T09:54:57.561+0100 DEBUG [cfgfile] cfgfile/cfgfile.go:193 Load config from file: /etc/metricbeat/modules.d/aws.yml
Feb 09 09:54:57 <host> metricbeat[10917]: 2021-02-09T09:54:57.562+0100 DEBUG [cfgfile] cfgfile/reload.go:213 Number of module configs found: 2
Feb 09 09:54:57 <host> metricbeat[10917]: 2021-02-09T09:54:57.562+0100 DEBUG [reload] cfgfile/list.go:63 Starting reload procedure, current runners: 0
Feb 09 09:54:57 <host> metricbeat[10917]: 2021-02-09T09:54:57.562+0100 DEBUG [reload] cfgfile/list.go:81 Start list: 2, Stop list: 0
Feb 09 09:54:57 <host> metricbeat[10917]: 2021-02-09T09:54:57.562+0100 DEBUG [get_aws_credentials] aws/credentials.go:40 Using access_key_id, secret_access_key and/or session_token for AWS credential
Feb 09 09:54:57 <host> metricbeat[10917]: 2021-02-09T09:54:57.562+0100 DEBUG [aws.billing] aws/aws.go:97 Metricset level config for period: 24h0m0s
Feb 09 09:54:57 <host> metricbeat[10917]: 2021-02-09T09:54:57.562+0100 DEBUG [aws.billing] aws/aws.go:98 Metricset level config for tags filter: []
Feb 09 09:54:57 <host> metricbeat[10917]: 2021-02-09T09:54:57.562+0100 WARN [aws.billing] aws/aws.go:99 extra charges on AWS API requests will be generated by this metricset
Feb 09 09:55:01 <host> metricbeat[10917]: 2021-02-09T09:55:01.395+0100 DEBUG [esclientleg] eslegclient/connection.go:364 GET https://local<host>:9200/ <nil>
Feb 09 09:55:01 <host> metricbeat[10917]: 2021-02-09T09:55:01.401+0100 INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(https://local<host>:9200)) established
Feb 09 09:55:02 <host> metricbeat[10917]: 2021-02-09T09:55:02.533+0100 DEBUG [elasticsearch] elasticsearch/client.go:230 PublishEvents: 4 events have been published to elasticsearch in 1.132275547s.
Feb 09 09:55:02 <host> metricbeat[10917]: 2021-02-09T09:55:02.533+0100 DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [0: 0, 4]
Feb 09 09:55:02 <host> metricbeat[10917]: 2021-02-09T09:55:02.533+0100 DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:4
Feb 09 09:55:02 <host> metricbeat[10917]: 2021-02-09T09:55:02.533+0100 DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack
Feb 09 09:55:25 <host> metricbeat[10917]: 2021-02-09T09:55:25.147+0100 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs
...

```

Unfortunately I still only get the billing metricset.

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [February 9, 2021, 5:22pm UTC](https://discuss.elastic.co/t/metricbeat-aws-module-billing-metricset/263223/7 "2021-02-09T17:22:28Z")

</div>

Hmm interesting!! What if you only have `ec2` metricset enabled in `aws.yml`like this:

```auto
- module: aws
  period: 5m
  metricsets:
    - ec2
  access_key_id: '<access_key>'
  secret_access_key: '<ecret_access_key>'

```

Do you see ec2 metrics in CloudWatch AWS portal?

---

<div class="post-metadata">

**Author:** ![Jurilz](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@Jurilz](https://discuss.elastic.co/u/Jurilz)\
**Post date:** [February 10, 2021, 12:38pm UTC](https://discuss.elastic.co/t/metricbeat-aws-module-billing-metricset/263223/8 "2021-02-10T12:38:13Z")

</div>

This config results in a empty `metricbeat-7.10.0-2021.02.10-000001` index with 0 documents.

This points to a lack of user permissions for my account, although my AWS Admin told me, that I have the following permissions:

```auto
ec2:DescribeInstances
ec2:DescribeRegions
cloudwatch:GetMetricData
cloudwatch:ListMetrics
sts:GetCallerIdentity
iam:ListAccountAliases
tag:getResources
sts:GetCallerIdentity
ce:GetCostAndUsage

```

which should sufice according to the AWS Module docu.

But in the Cloudwatch Portal I only see the metrics for:

Services: CloudWatch, Resource: ListMetrics, Type: API, Class: None, Metric name: CallCount

Services: CloudWatch, Resource: DescribeAlarm, Type: API, Class: None, Metric name: CallCount

---

<div class="post-metadata">

**Author:** ![Jurilz](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@Jurilz](https://discuss.elastic.co/u/Jurilz)\
**Post date:** [February 10, 2021, 12:47pm UTC](https://discuss.elastic.co/t/metricbeat-aws-module-billing-metricset/263223/9 "2021-02-10T12:47:32Z")

</div>

The billing metricset tells me only about the cost of the AWS Cost Explorer and the AmazonCloudWatch services, but not about the Hosting costs. ![aws.billing.group_by.SERVIC](https://us1.discourse-cdn.com/elastic/original/3X/5/a/5a78820cad94f8682af904a6caac490a08be50ed.png)

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [February 10, 2021, 6:04pm UTC](https://discuss.elastic.co/t/metricbeat-aws-module-billing-metricset/263223/10 "2021-02-10T18:04:42Z")

</div>

![Screen Shot 2021-02-10 at 10.07.10 AM](https://us1.discourse-cdn.com/elastic/original/3X/2/4/24223003fecc6f8b2630f9fac7a202a8395902ec.png)  
Sorry I didn't make it clear, I would check here to make sure there are EC2 metrics.

Thanks for the info on billing! What does the hosting cost look like on AWS Cost Explorer portal? Is it under grouped by usage type? Sorry I'm not familiar with the hosting cost 🙂

---

<div class="post-metadata">

**Author:** ![Jurilz](https://avatars.discourse-cdn.com/v4/letter/j/278dde/32.png) [@Jurilz](https://discuss.elastic.co/u/Jurilz)\
**Post date:** [February 11, 2021, 12:27pm UTC](https://discuss.elastic.co/t/metricbeat-aws-module-billing-metricset/263223/11 "2021-02-11T12:27:42Z")

</div>

Thank you again for your persistence on this topic.

I seem to not have any permission to see EC2 Metrics or any billing information in the AWS Cost Explorer.

This is how my Clodwatch dashboard looks like.

 ![cloudwatch](https://us1.discourse-cdn.com/elastic/original/3X/2/3/232958d65b0e2778cd5fe1252af0588d3305d0d0.png)

I apologize for everything is in german.

AWS Cost Explorer tells me:

"You are not authorized to perform this operation.

The IAM policy for the current user does not allow access to this page. IAM policies must be enabled to allow access to the Billing and Expense Management console."

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 11, 2021, 2:28pm UTC](https://discuss.elastic.co/t/metricbeat-aws-module-billing-metricset/263223/12 "2021-03-11T14:28:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.