AWS metricbeat unable to load data in kibana

Elastic Stack Beats
1

Hi ,

I am using metricbeat 7.14. IAM policy given for AWS billing and AWS.yml config file is as follows as stated in

AWS billing metricset | Metricbeat Reference [7.16] | Elastic

Step I followed to start aws and flow aws billing data to kibana are as follows.

S1>Make changes in metricbeat.yml (changed cloud id,user/password)
S2>Enable AWS
S3>Make changes in AWS.yml
S4>Start metricbeat, but data didn't flow.

When I checked the status of metricbeat, i am getting failed message.

sudo service metricbeat status
● metricbeat.service - Metricbeat is a lightweight shipper for metrics.
   Loaded: loaded (/usr/lib/systemd/system/metricbeat.service; disabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Mon 2021-09-06 07:33:16 UTC; 23s ago
     Docs: https://www.elastic.co/beats/metricbeat
  Process: 24312 ExecStart=/usr/share/metricbeat/bin/metricbeat --environment systemd $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS (code=exited, status=1/FAILURE)
 Main PID: 24312 (code=exited, status=1/FAILURE)

Sep 06 07:33:16 dev systemd[1]: Unit metricbeat.service entered failed state.
Sep 06 07:33:16 dev systemd[1]: metricbeat.service failed.
Sep 06 07:33:16 dev systemd[1]: metricbeat.service holdoff time over, scheduling restart.
Sep 06 07:33:16 dev systemd[1]: Stopped Metricbeat is a lightweight shipper for metrics..
Sep 06 07:33:16 dev systemd[1]: start request repeated too quickly for metricbeat.service
Sep 06 07:33:16 dev systemd[1]: Failed to start Metricbeat is a lightweight shipper for metrics..
Sep 06 07:33:16 dev systemd[1]: Unit metricbeat.service entered failed state.
Sep 06 07:33:16 dev systemd[1]: metricbeat.service failed.

As metricbeat is not starting then I disable aws module and try starting metricbeat again.Then Metrcibeat gets started.
Then I enable aws module but still aws data is not flowing to the kibana,

Pls let me know if i am making mistake in running the step or in aws.yml file.

sudo service metricbeat status
● metricbeat.service - Metricbeat is a lightweight shipper for metrics.
   Loaded: loaded (/usr/lib/systemd/system/metricbeat.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2021-09-06 07:36:47 UTC; 7s ago
     Docs: https://www.elastic.co/beats/metricbeat
 Main PID: 27599 (metricbeat)
    Tasks: 10
   Memory: 55.5M
   CGroup: /system.slice/metricbeat.service
           └─27599 /usr/share/metricbeat/bin/metricbeat --environment systemd -c /etc/metricbeat/metricbeat.yml --path.home /usr/share/metricbeat --path.config /etc/...

Sep 06 07:36:48 dev metricbeat[27599]: 2021-09-06T07:36:48.773Z        INFO        [index-management]        idxmgmt/std.go:261        Auto ILM enable success.
Sep 06 07:36:48 dev metricbeat[27599]: 2021-09-06T07:36:48.807Z        INFO        [index-management.ilm]        ilm/std.go:160        ILM policy metricbea...s already.
Sep 06 07:36:48 dev metricbeat[27599]: 2021-09-06T07:36:48.807Z        INFO        [index-management]        idxmgmt/std.go:401        Set setup.template.n...s enabled.
Sep 06 07:36:48 dev metricbeat[27599]: 2021-09-06T07:36:48.807Z        INFO        [index-management]        idxmgmt/std.go:406        Set setup.template.p...s enabled.
Sep 06 07:36:48 dev metricbeat[27599]: 2021-09-06T07:36:48.807Z        INFO        [index-management]        idxmgmt/std.go:440        Set settings.index.l...s enabled.
Sep 06 07:36:48 dev metricbeat[27599]: 2021-09-06T07:36:48.807Z        INFO        [index-management]        idxmgmt/std.go:444        Set settings.index.l...s enabled.
Sep 06 07:36:48 dev metricbeat[27599]: 2021-09-06T07:36:48.843Z        INFO        template/load.go:111        Template "metricbeat-7.14.0" already exists ...erwritten.
Sep 06 07:36:48 dev metricbeat[27599]: 2021-09-06T07:36:48.843Z        INFO        [index-management]        idxmgmt/std.go:297        Loaded index template.
Sep 06 07:36:48 dev metricbeat[27599]: 2021-09-06T07:36:48.883Z        INFO        [index-management.ilm]        ilm/std.go:121        Index Alias metricbe...s already.
Sep 06 07:36:48 dev metricbeat[27599]: 2021-09-06T07:36:48.884Z        INFO        [publisher_pipeline_output]        pipeline/output.go:151        Connect...stablished
Hint: Some lines were ellipsized, use -l to show in full.

Aws.yml

- module: aws
   period: 60s
  metricsets:
    - billing
  cost_explorer_config:
    group_by_dimension_keys:
      - "AZ"
      - "INSTANCE_TYPE"
      - "SERVICE"
      - "LINKED_ACCOUNT"
    group_by_tag_keys:
      - "aws:createdBy"
  access_key_id: '${AWS_ACCESS_KEY_ID:"<>"}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY:"< >"}'
2

Hi @Divyank_Mahalle !

Do you see anything related in Metricbeat's logs? If it is an issue with your configuration you should be able to see the error in Metricbear's logs.

C.

3

Hi Chris,
I am not able to seeing any error in metricbeat logs.

I checked with IAM polciy, its applies correctly as per 7.14.
Few days back I am able to aws logs to kibana with ES version 7.8,post upgradation aws logs are not loading but system metrics data is loading successfully in metricbeat-* index in old and new ES 7.8 and 7.14 version.
I am not getting where I am doing mistake.All config is same just 1 IAM policy organizationaccount is added for 7.14 version.

4

@Kaiyan_Sheng would you mild checking if there is anything obvious here?

5

Can you post the Metricbeat logs. If its not showing much can you try to manually start Metricbeat from the CLI using the -e flag to run in debug mode. That will output a lot more data that we can use to diagnosis the issue.

7

Current Logs-Now no errors and aws details are also getting loaded in logs but still aws data is not flowing.
Pls let me know any changes required in my aws.yml config or any logs required.

DEBUG [cfgfile] cfgfile/reload.go:213 Number of module configs found: 0

sudo metricbeat -e -d "*"

2021-09-10T08:42:04.167-0400    INFO    instance/beat.go:665    Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
2021-09-10T08:42:04.167-0400    DEBUG   [beat]  instance/beat.go:723    Beat metadata path: /var/lib/metricbeat/meta.json
2021-09-10T08:42:04.167-0400    INFO    instance/beat.go:673    Beat ID: 0234d09a-49fa-471a-8193-62ca98cbdd15
2021-09-10T08:42:04.171-0400    DEBUG   [add_cloud_metadata]    add_cloud_metadata/providers.go:165     add_cloud_metadata: received disposition for aws after 1.757165ms. result=[provider:aws, error=<nil>, metadata={"cloud":{"account":{"id":"<>"},"availability_zone":"us-east-1a","image":{"id":"<>"},"instance":{"id":"<>"},"machine":{"type":"<>"},"provider":"aws","region":"us-east-1","service":{"name":"EC2"}}}]
2021-09-10T08:42:04.171-0400    DEBUG   [add_cloud_metadata]    add_cloud_metadata/providers.go:131     add_cloud_metadata: fetchMetadata ran for 1.844447ms
2021-09-10T08:42:04.171-0400    INFO    [add_cloud_metadata]    add_cloud_metadata/add_cloud_metadata.go:105    add_cloud_metadata: hosting provider type detected as aws, metadata={"cloud":{"account":{"id":"<>"},"availability_zone":"us-east-1a","image":{"id":"<>"},"instance":{"id":"i-030b87c242e566df2"},"machine":{"type":"t3.large"},"provider":"aws","region":"us-east-1","service":{"name":"EC2"}}}
2021-09-10T08:42:04.171-0400    INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2021-09-10T08:42:04.172-0400    INFO    [beat]  instance/beat.go:1014   Beat info       {"system_info": {"beat": {"path": {"config": "/etc/metricbeat", "data": "/var/lib/metricbeat", "home": "/usr/share/metricbeat", "logs": "/var/log/metricbeat"}, "type": "metricbeat", "uuid": "0234d09a-49fa-471a-8193-62ca98cbdd15"}}}
2021-09-10T08:42:04.172-0400    INFO    [beat]  instance/beat.go:1023   Build info      {"system_info": {"build": {"commit": "e127fc31fc6c00fdf8649808f9421d8f8c28b5db", "libbeat": "7.14.0", "time": "2021-07-29T21:09:05.000Z", "version": "7.14.0"}}}
2021-09-10T08:42:04.172-0400    INFO    [beat]  instance/beat.go:1026   Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.16.6"}}}
2021-09-10T08:42:04.172-0400    INFO    [beat]  instance/beat.go:1030   Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-02-22T08:56:08-05:00","containerized":false,"name":"dev-0","ip":["127.0.0.1/8","::1/128","172.30.0.63/24","fe80::c8a:d3ff:fe97:b675/64"],"kernel_version":"5.4.0-1037-aws","mac":["0e:8a:d3:97:b6:75"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"18.04.5 LTS (Bionic Beaver)","major":18,"minor":4,"patch":5,"codename":"bionic"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"ec211df81d25fc20b5021dac6516d70c"}}}
2021-09-10T08:42:04.173-0400    INFO    instance/beat.go:309    Setup Beat: metricbeat; Version: 7.14.0
2021-09-10T08:42:04.173-0400    DEBUG   [beat]  instance/beat.go:335    Initializing output plugins
2021-09-10T08:42:04.173-0400    INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'metricbeat-7.14.0' as ILM is enabled.
2021-09-10T08:42:04.173-0400    INFO    [esclientleg]   eslegclient/connection.go:100   elasticsearch url: <>
2021-09-10T08:42:04.173-0400    DEBUG   [publisher]     pipeline/consumer.go:148        start pipeline event consumer
2021-09-10T08:42:04.173-0400    INFO    [publisher]     pipeline/module.go:113  Beat name: dev-0
2021-09-10T08:42:04.220-0400    INFO    instance/beat.go:473    metricbeat start running.
2021-09-10T08:42:04.220-0400    DEBUG   [cfgfile]       cfgfile/reload.go:132   Checking module configs from: /etc/metricbeat/modules.d/*.yml
2021-09-10T08:42:04.221-0400    DEBUG   [cfgfile]       cfgfile/cfgfile.go:193  Load config from file: /etc/metricbeat/modules.d/aws.yml
2021-09-10T08:42:04.221-0400    DEBUG   [cfgfile]       cfgfile/reload.go:146   Number of module configs found: 0
2021-09-10T08:42:04.220-0400    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2021-09-10T08:42:04.221-0400    INFO    cfgfile/reload.go:164   Config reloader started
2021-09-10T08:42:04.221-0400    DEBUG   [cfgfile]       cfgfile/reload.go:194   Scan for new config files
2021-09-10T08:42:04.221-0400    DEBUG   [cfgfile]       cfgfile/cfgfile.go:193  Load config from file: /etc/metricbeat/modules.d/aws.yml
2021-09-10T08:42:04.221-0400    DEBUG   [cfgfile]       cfgfile/reload.go:213   Number of module configs found: 0
2021-09-10T08:42:04.221-0400    DEBUG   [reload]        cfgfile/list.go:63      Starting reload procedure, current runners: 0
2021-09-10T08:42:04.221-0400    DEBUG   [reload]        cfgfile/list.go:81      Start list: 0, Stop list: 0
2021-09-10T08:42:04.221-0400    INFO    cfgfile/reload.go:224   Loading of config files completed.

aws.yml config-

metricbeat.modules:
- module: aws
  period: 60s
  metricsets:
     - billing
  access_key_id: '${AWS_ACCESS_KEY_ID:"<>"}'
  secret_access_key: '${AWS_SECRET_ACCESS_KEY:"<>"}'

While I changed aws.yml confg its throwing error.

instance/beat.go:989 Exiting: 1 error: error creating aws metricset: failed DescribeRegions: UnauthorizedOperation: You are not authorized to perform this operation.

2021-09-10T10:40:13.350-0400    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2021-09-10T10:40:13.350-0400    INFO    instance/beat.go:473    metricbeat start running.
2021-09-10T10:40:13.350-0400    DEBUG   [cfgfile]       cfgfile/reload.go:132   Checking module configs from: /etc/metricbeat/modules.d/*.yml
2021-09-10T10:40:13.351-0400    DEBUG   [cfgfile]       cfgfile/cfgfile.go:193  Load config from file: /etc/metricbeat/modules.d/aws.yml
2021-09-10T10:40:13.351-0400    DEBUG   [cfgfile]       cfgfile/reload.go:146   Number of module configs found: 1
2021-09-10T10:40:13.351-0400    DEBUG   [getAccessKeys] aws/credentials.go:69   Using access keys for AWS credential
2021-09-10T10:40:13.351-0400    DEBUG   [aws.billing]   aws/aws.go:89   aws config endpoint =
2021-09-10T10:40:13.351-0400    DEBUG   [aws.billing]   aws/aws.go:99   Metricset level config for period: 12h0m0s
2021-09-10T10:40:13.351-0400    DEBUG   [aws.billing]   aws/aws.go:100  Metricset level config for tags filter: []
2021-09-10T10:40:13.351-0400    WARN    [aws.billing]   aws/aws.go:101  extra charges on AWS API requests will be generated by this metricset
2021-09-10T10:40:13.415-0400    DEBUG   [aws.billing]   aws/aws.go:131  AWS Credentials belong to account ID: <>
2021-09-10T10:40:13.445-0400    DEBUG   [aws.billing]   aws/aws.go:193  AWS Credentials belong to account name:
2021-09-10T10:42:46.051-0400    ERROR   instance/beat.go:989    Exiting: 1 error: error creating aws metricset: failed DescribeRegions: UnauthorizedOperation: You are not authorized to perform this operation.

Aws.yml config-(I removed first line metricbeat.modules in it.

- module: aws
  period: 60s
  metricsets:
    - billing
  access_key_id: "< >"
  secret_access_key: "< >"
8

Seems like you didn't give ur user/keys the correct IAM roles/permissions

9

Hi

Thanks for your valuable inputs, billing data successfully flowing through kibana after adding "ec2:DescribeRegions" policy.

But in 7.14 metrcibeat official documentation this policy is not present.

My working IAM policy looks like this-

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "organizations:ListAccounts",
                "tag:GetResources",
                "cloudwatch:GetMetricData",
                "iam:ListAccountAliases",
                "ce:GetCostAndUsage",
                "sts:GetCallerIdentity",
                "cloudwatch:ListMetrics",
                "ec2:DescribeRegions"
            ],
            "Resource": "*"
        }
    ]
}
10

This is what I see in the docs., AWS usage metricset | Metricbeat Reference [7.14] | Elastic. if you think it's missing permissions, feel free to open a GitHub issue.

11

Ah yes, in usage metricset "ec2:DescribeRegions" is present.

As I was checking for billing metricset for aws billing data, "ec2:DescribeRegions" is not present.

But anyways, Thanks once again.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.