AWS metricbeat config, Access Denied

Elastic Stack Beats
1

Hello,

I am trying to configure Metricbeat to collect AWS metrics. I have tried configuring it to use credential profile and even explicity setting the access keys inside the yml. It keeps saying I dont have enough permissions to DescribeRegions. I tried giving the user higher permissions which includes that and still the same issue. Any help on this would be great. I am also using the latest version of Elastic/Beats.

- module: aws
  period: 1m
  metricsets:
    - elb
    - usage
  access_key_id: ''
  secret_access_key: ''
  credential_profile_name: default
  regions:
    - us-east-1
- module: aws
  period: 5m
  access_key_id: ''"
  secret_access_key: ''"
  credential_profile_name: default
  metricsets:
    - cloudwatch
  metrics:
    - namespace: AWS/EC2
      #name: ["CPUUtilization", "DiskWriteOps"]
      #dimensions:
      #  - name: InstanceId
      #    value: i-0686946e22cf9494a
      #statistic: ["Average", "Maximum"]
  regions:
    - us-east-1
- module: aws
  period: 5m
  access_key_id: ''
  secret_access_key: ''
  metricsets:
    - ebs
    - ec2
    - sns
    - sqs
    - rds
  regions:
    - us-east-1
- module: aws
  period: 12h
  access_key_id: ''
  secret_access_key: ''
  metricsets:
    - billing
  regions:
    - us-east-1
- module: aws
  period: 24h
  access_key_id: ''
  secret_access_key: ''
  metricsets:
    - s3_daily_storage
    - s3_request

Thanks,
Kenneth

2

Weird issue, I can send billing metrics no problem but all other resources that require DescribeRegion fail. Could this be a possible bug? I gave my user Ec2FullAccess and also explicitly gave that permission.

3

Was anyone able to take a look at this?

4

Hi @kmroz, thanks for posting your question here! Could you by any chance to copy your policy JSON file here please?

You can find it in IAM console on AWS. Here is what mine looks like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor3",
            "Effect": "Allow",
            "Action": [
                "tag:GetResources",
                "ec2:DescribeInstances",
                "cloudwatch:GetMetricData",
                "ec2:DescribeRegions",
                "iam:ListAccountAliases",
                "sts:GetCallerIdentity",
                "cloudwatch:ListMetrics"
            ],
            "Resource": "*"
        }
    ]
}
5

Also if you want to verify if the credentials you use in shared credential file is good, you can try some basic AWS CLIs locally like https://docs.aws.amazon.com/cli/latest/reference/iam/list-account-aliases.html

6 
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeRegions",
                "cloudwatch:GetMetricData",
                "cloudwatch:ListMetrics",
                "tag:getResources",
                "sts:GetCallerIdentity",
                "iam:ListAccountAliases",
                "ec2:DescribeInstances",
                "rds:DescribeDBInstances",
                "sqs:ListQueues"
            ],
            "Resource": "*"
        }
    ]
}
7

also I am able to confirm via the CLI i can describe regions.

8

Thank you!!! The permission seems fine hmmm Could you try start Metricbeat with config file aws.yml only includes ec2 metricset and copy paste your whole Metricbeat log to here please?

- module: aws
  period: 5m
  credential_profile_name: <credential-profile-name>
  metricsets:
    - ec2

Thanks again!

  • Kaiyan
9 
1 error: error creating aws metricset: Failed DescribeRegions: UnauthorizedOperation: You are not authorized to perform this operation.

Is the error i have recieved.

10

I just downloaded the latest Metricbeat (version 7.6.2) and used config:

- module: aws
  period: 5m
  credential_profile_name: elastic-beats
  metricsets:
    - ec2

I was able to collect metrics from EC2 instances.

When you tested with AWS CLI, what credential did you use? If you copy these credentials directly into the aws.yml config, do you also get the same error?

- module: aws
  period: 5m
  access_key_id: <'access key id'>
  secret_access_key: <'secret access key'>
  session_token: <'if you have a session token'>
  metricsets:
    - ec2
11

Hey ,

Yes , i have tried both pasting in the config and using the profile_name. It gives me the same error.

12

so, it seems after doing a daemon-reload ec2 started to work. I am now running into the issue with S3 saying i need to DescrbeRegions

13

weirdly enough had to add regions sections to this to get it working. I think i am running now. Thank you

1 Like
14

oh no... I can't think of anything if the config is right though hmmm I can't reproduce it on my side :frowning_face:

If you run ./metricbeat -e -d '*' to enable debug log, will you able to see any other information might be useful?

15

Ahh great! I didn't see your recent comments! Thanks for letting me know!!

1 Like
16

No problem, thanks for helping out!

1 Like
17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.