# Custom Cloudwatch Metrics

**URL:** <https://discuss.elastic.co/t/custom-cloudwatch-metrics/221324>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [February 27, 2020, 8:29pm UTC](https://discuss.elastic.co/t/custom-cloudwatch-metrics/221324 "2020-02-27T20:29:03Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![AddChickpeas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/addchickpeas/32/44414_2.png) [@AddChickpeas](https://discuss.elastic.co/u/AddChickpeas)\
**Post date:** [February 27, 2020, 8:29pm UTC](https://discuss.elastic.co/t/custom-cloudwatch-metrics/221324/1 "2020-02-27T20:29:04Z")

</div>

Hello,

Using v7.6.0

I'm attempting to collect some custom metrics using metricbeat, but not having any luck. I have tried pretty much every combination of the below possible

```
 - module: aws
  period: 60s
  access_key_id: '${AWS_ACCESS_KEY_ID}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY}'
  tags: "pre"
  metricsets:
    - cloudwatch
  metrics:            
    - namespace: ELK  
      name: ["ELKResponseTime"]
      statistic: ["Average"]          
      dimensions:
          - name: ELKResponseTime
            value: ELK    

```

I am able to successfully pull from cloudwatching using `get-metric-data` using the same info.

```
{
        "MetricDataQueries": [
    {
        "Id": "m1",
        "MetricStat": {
                "Metric": {
                        "Namespace": "ELK",
                        "MetricName": "ELKResponseTime",
                        "Dimensions" : [{
                                "Name": "ELKResponseTime",
                                "Value": "ELK"
                                }]
                },
                "Period": 60,
                "Stat": "Average"
                        }
                }
    ],
    "StartTime": "2020-02-27T14:00:0000",
    "EndTime": "2020-02-27T14:15:0000"
}

```

response

```
{
    "Messages": [],
    "MetricDataResults": [
        {
            "Id": "m1",
            "Timestamps": [
                "2020-02-27T14:14:00Z",
                "2020-02-27T14:13:00Z",
                "2020-02-27T14:12:00Z",
                "2020-02-27T14:11:00Z",
                "2020-02-27T14:10:00Z",
                "2020-02-27T14:09:00Z",
                "2020-02-27T14:08:00Z",
                "2020-02-27T14:07:00Z",
                "2020-02-27T14:06:00Z",
                "2020-02-27T14:05:00Z",
                "2020-02-27T14:04:00Z",
                "2020-02-27T14:03:00Z",
                "2020-02-27T14:02:00Z",
                "2020-02-27T14:01:00Z",
                "2020-02-27T14:00:00Z"
            ],
            "StatusCode": "Complete",
            "Label": "ELKResponseTime",
            "Values": [
                0.09296989440917969,
                0.0331873893737793,
                0.11049914360046387,
                0.03397560119628906,
                0.045053720474243164,
                0.04449105262756348,
                0.0390627384185791,
                0.03135561943054199,
                0.0417938232421875,
                0.04479169845581055,
                0.12064003944396973,
                0.11304378509521484,
                0.03579378128051758,
                0.026007890701293945,
                0.19768404960632324
            ]
        }
    ]
}
```

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [February 28, 2020, 9:25am UTC](https://discuss.elastic.co/t/custom-cloudwatch-metrics/221324/2 "2020-02-28T09:25:10Z")

</div>

Hi!

Not sure if this is supported 🤔. Maybe @Kaiyan_Sheng could verify.

C.

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [February 28, 2020, 2:44pm UTC](https://discuss.elastic.co/t/custom-cloudwatch-metrics/221324/3 "2020-02-28T14:44:35Z")

</div>

@AddChickpeas Thanks for posting here! I'm gonna try reproduce it today to see what happens with custom metrics. Cloudwatch metricset is using `ListMetrics` and `GetMetricData` two API calls. Good to know that get-metric-data API is working. Do you know if `ListMetrics` works with `namespace=ELK`?

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [February 28, 2020, 4:32pm UTC](https://discuss.elastic.co/t/custom-cloudwatch-metrics/221324/4 "2020-02-28T16:32:35Z")

</div>

Hi @AddChickpeas, I just created some custom metrics and stored in CloudWatch namespace called `Custom`. With metricbeat `aws` module enabled with `aws.yml`:

```auto
- module: aws
  period: 5m
  credential_profile_name: elastic-beats
  regions:
    - us-east-1
  metricsets:
    - cloudwatch
  metrics:
    - namespace: Custom

```

I was able to collect these metrics from Custom namespace:

```auto
{
  "_index": "metricbeat-8.0.0-2020.02.28-000001",
  "_type": "_doc",
  "_id": "yVibjHABNNjmvAqrcSsh",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2020-02-28T16:22:39.031Z",
    "host": {
      "architecture": "x86_64",
      "os": {
        "name": "Mac OS X",
        "kernel": "17.7.0",
        "build": "17G10021",
        "platform": "darwin",
        "version": "10.13.6",
        "family": "darwin"
      },
      "id": "9C7FAB7B-29D1-5926-8E84-158A9CA3E25D",
      "name": "KaiyanMacBookPro",
      "ip": [
        "fe80::18da:9553:5e03:fd2d",
        "192.168.0.5",
        "fe80::6c96:28ff:fe6d:fcdc",
        "fe80::1b53:9dee:f305:e2c7",
        "fe80::aede:48ff:fe00:1122"
      ],
      "mac": [
        "f2:18:98:75:78:55",
        "f0:18:98:75:78:55",
        "02:18:98:75:78:55",
        "6e:96:28:6d:fc:dc",
        "c6:00:e4:40:f6:04",
        "c6:00:e4:40:f6:01",
        "c6:00:e4:40:f6:00",
        "c6:00:e4:40:f6:05",
        "c6:00:e4:40:f6:01",
        "ac:de:48:00:11:22"
      ],
      "hostname": "KaiyanMacBookPro"
    },
    "agent": {
      "type": "metricbeat",
      "ephemeral_id": "5f9d5e70-b0e3-4c0b-a7cf-0fbef5fc92cb",
      "hostname": "KaiyanMacBookPro",
      "id": "23fd6ee9-2255-43f3-9565-78a051337c04",
      "version": "8.0.0"
    },
    "aws": {
      "custom": {
        "metrics": {
          "memory-usage": {
            "max": 83.05,
            "min": 82.94,
            "sum": 414.81,
            "count": 5,
            "avg": 82.962
          },
          "Tcp_connections": {
            "avg": 56,
            "max": 56,
            "min": 56,
            "sum": 280,
            "count": 5
          },
          "IO_WAIT": {
            "count": 5,
            "avg": 0.01,
            "max": 0.01,
            "min": 0.01,
            "sum": 0.05
          },
          "TCP_connection_on_port_80": {
            "avg": 3,
            "max": 3,
            "min": 3,
            "sum": 15,
            "count": 5
          },
          "No_of_users": {
            "avg": 1,
            "max": 1,
            "min": 1,
            "sum": 5,
            "count": 5
          }
        }
      },
      "cloudwatch": {
        "namespace": "Custom"
      },
      "dimensions": {
        "Instance": "i-05381e495aa0f95ef"
      }
    },
    "event": {
      "dataset": "aws.cloudwatch",
      "module": "aws",
      "duration": 651057530
    },
    "metricset": {
      "period": 300000,
      "name": "cloudwatch"
    },
    "service": {
      "type": "aws"
    },
    "cloud": {
      "provider": "aws",
      "region": "us-east-1",
      "account": {
        "name": "elastic-beats",
        "id": "428152502467"
      }
    },
    "ecs": {
      "version": "1.4.0"
    }
  },
  "fields": {
    "@timestamp": [
      "2020-02-28T16:22:39.031Z"
    ]
  },
  "sort": [
    1582906959031
  ]
}

```

Do you see any error message in Metricbeat log? Maybe you can try with a simpler config first, like:

```auto
 - module: aws
  period: 60s
  access_key_id: '${AWS_ACCESS_KEY_ID}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY}'
  metricsets:
    - cloudwatch
  metrics:            
    - namespace: ELK 

```

---

<div class="post-metadata">

**Author:** ![AddChickpeas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/addchickpeas/32/44414_2.png) [@AddChickpeas](https://discuss.elastic.co/u/AddChickpeas)\
**Post date:** [February 28, 2020, 8:16pm UTC](https://discuss.elastic.co/t/custom-cloudwatch-metrics/221324/5 "2020-02-28T20:16:49Z")

</div>

Hi Kaiyan,

Thanks for getting back to me.

I tested list-metrics and it is returning logs. Looks like the person that added them hasn't really cleaned it up all that much, but the metrics are there. There are a bunch of duplicates so not sure if that could be effecting things. I'll see if they added any others I can test with.

```
{
"Metrics": [       
    {
        "Dimensions": [
            {
                "Value": "ELK",
                "Name": "ELKResponseTime"
            }
        ],
        "Namespace": "ELK",
        "MetricName": "ELKResponseTime"
    },
    {
        "Dimensions": [
            {
                "Value": "ELK",
                "Name": "ELKResponseError"
            }
        ],
        "Namespace": "ELK",
        "MetricName": "ELKResponseError"
    },
    {
        "Dimensions": [
            {
                "Value": "ELK",
                "Name": "ELKResponseError"
            }
        ],
        "Namespace": "ELK",
        "MetricName": "ELKResponseTime"
    },
    {
        "Dimensions": [
            {
                "Value": "ELK",
                "Name": "ELKResponseTime"
            }
        ],
        "Namespace": "ELK",
        "MetricName": "ELKResponseError"
    }
]
}

```

I updated the config just to include this,. but no luck

```
metricbeat.modules:
- module: aws
  period: 60s
  access_key_id: '${AWS_ACCESS_KEY_ID_DATADEV}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY_DATADEV}'
  tags: ["dev"]        
  metricsets:
    - cloudwatch
  metrics:
    - namespace: ELK 

```

Nothing in logs either even with debug on:

```
2020-02-28T20:12:55.515Z INFO [monitoring log/log.go:118 Starting metrics logging every 30s
2020-02-28T20:12:55.515Z INFO instance/beat.go:439 metricbeat start running.
2020-02-28T20:12:55.516Z DEBUG [module module/wrapper.go:120 Starting Wrapper[name=aws, len(metricSetWrappers)=1]
2020-02-28T20:12:55.516Z DEBUG [module module/wrapper.go:174 aws/cloudwatch will start after 7.222403846s
2020-02-28T20:13:02.738Z DEBUG [module module/wrapper.go:182 Starting metricSetWrapper[module=aws, name=cloudwatch, host=]
2020-02-28T20:13:25.526Z INFO [monitoring log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":100,"time":{"ms":109}},"total":{"ticks":290,"time":{"ms":304},"value":290},"user":{"ticks":190,"time":{"ms":195}}},"handles":{"limit":{"hard":65536,"soft":65536},"open":7},"info":{"ephemeral_id":"8c72a0ab-8741-4cd4-97d3-2405f89f5365","uptime":{"ms":36306}},"memstats":{"gc_next":14784800,"memory_alloc":11824680,"memory_total":29226752,"rss":75718656},"runtime":{"goroutines":22}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0}}},"system":{"cpu":{"cores":4},"load":{"1":11.46,"15":13.99,"5":13.32,"norm":{"1":2.865,"15":3.4975,"5":3.33}}}}}}
```

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [February 28, 2020, 8:20pm UTC](https://discuss.elastic.co/t/custom-cloudwatch-metrics/221324/6 "2020-02-28T20:20:41Z")

</div>

Cool, then this cloudwatch metricset should work. Could you test it with a simpler config like this one please?

```auto
 - module: aws
  period: 60s
  access_key_id: '${AWS_ACCESS_KEY_ID}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY}'
  metricsets:
    - cloudwatch
  metrics:            
    - namespace: ELK 

```

Also, if it failed, could you post some logs please. That would be helpful too! Thanks!

---

<div class="post-metadata">

**Author:** ![AddChickpeas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/addchickpeas/32/44414_2.png) [@AddChickpeas](https://discuss.elastic.co/u/AddChickpeas)\
**Post date:** [February 28, 2020, 8:22pm UTC](https://discuss.elastic.co/t/custom-cloudwatch-metrics/221324/7 "2020-02-28T20:22:30Z")

</div>

Please see my edit in the previous post. I edited to add that information because I submitted it early by accident.

I tested with that config, but no luck.

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [February 28, 2020, 8:27pm UTC](https://discuss.elastic.co/t/custom-cloudwatch-metrics/221324/8 "2020-02-28T20:27:48Z")

</div>

Thanks! Could you remove `tags: ["dev"] `and rerun Metricbeat please? I just saw this in the config hmmm  
Are you trying to use tags to filter?

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [February 28, 2020, 8:29pm UTC](https://discuss.elastic.co/t/custom-cloudwatch-metrics/221324/9 "2020-02-28T20:29:46Z")

</div>

`tags` as filter for cloudwatch metricset is poorly documented (not documented ☹ ). I will push a PR to add this for sure. Sorry about that! Here is the PR for adding tags as filter and the PR description has some example on how to use it: [https://github.com/elastic/beats/pull/13758](https://github.com/elastic/beats/pull/13758)

---

<div class="post-metadata">

**Author:** ![AddChickpeas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/addchickpeas/32/44414_2.png) [@AddChickpeas](https://discuss.elastic.co/u/AddChickpeas)\
**Post date:** [February 28, 2020, 8:47pm UTC](https://discuss.elastic.co/t/custom-cloudwatch-metrics/221324/10 "2020-02-28T20:47:49Z")

</div>

I was just adding tags into the event like you can with other beats. The \_doc ends up having a `tags` field with `pre` as the value.

```
metricset.name cloudwatch
metricset.period 60,000
service.type aws
tags pre

```

I'll give it a try without and get back to you. We're making some changes so I'll have to wait until it actually starts outputting the custom metrics again.

I will give that `tags` as filter a try! I tried it at one point, but didn't have much luck. I ended up just listing items I wanted to pull metrics on manually.

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [February 28, 2020, 8:51pm UTC](https://discuss.elastic.co/t/custom-cloudwatch-metrics/221324/11 "2020-02-28T20:51:11Z")

</div>

Ahh sorry I misunderstood. The PR I linked is to use tags as filters for collecting metrics from CloudWatch. If you want to collect all metrics from ELK namespace, please ignore my comment about filtering with tags 😬 Thanks!

---

<div class="post-metadata">

**Author:** ![AddChickpeas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/addchickpeas/32/44414_2.png) [@AddChickpeas](https://discuss.elastic.co/u/AddChickpeas)\
**Post date:** [February 28, 2020, 9:15pm UTC](https://discuss.elastic.co/t/custom-cloudwatch-metrics/221324/12 "2020-02-28T21:15:51Z")

</div>

So I think the issue was with the configuration of the custom metrics themselves. I switched to using a different custom metric and it is working as expected. Not really sure what the difference between them is.

Thanks for your help!

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [February 28, 2020, 9:55pm UTC](https://discuss.elastic.co/t/custom-cloudwatch-metrics/221324/13 "2020-02-28T21:55:07Z")

</div>

Interesting! Good to know! Thanks for all the info!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 27, 2020, 9:55pm UTC](https://discuss.elastic.co/t/custom-cloudwatch-metrics/221324/14 "2020-03-27T21:55:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.