Issues configuring AWS Module for MetricBeat

[ozonshak]

Hello. I'm attempting to set up the AWS module for the first time with MetricBeat. I've got the module enabled, got past the authentication errors I was seeing (created an AWS user with full "ReadOnlyPolicy"), and updated the aws.yml file. I am seeing a little bit of information, which is telling me that my account authorization is working. However, the only 2 services I'm seeing statistics for is EC2 and S3. I'm using the Kibana dashboards I loaded from Elastic for AWS. Here are the issues I'm seeing:

EC2 - only seeing 5 instances - choices seem random
EBS - no statistics at all
RDS - no statistics at all
Usage - seeing minor stats for EC2 and Cloudwatch, but dashboard is saying:
Could not locate that index-pattern-field (id: aws.usage.metrics.ResourceCount.sum)
Could not locate that index-pattern-field (id: aws.usage.metrics.CallCount.sum)

So, I'm definitely connected, but I'm getting very inconsistent data. Here is what my YML looks like:

- module: aws
  period: 5m
  metricsets:
    - cloudwatch
  metrics:
    - namespace: "*"
    #- namespace: AWS/EC2
      #name: ["CPUUtilization", "DiskWriteOps"]
    #  tags.resource_type_filter: ec2:instance
    #- namespace: AWS/EBS
    #- namespace: AWS/ELB
    #- namespace: AWS/Billing
    #- namespace: AWS/DX
    #- namespace: AWS/ElasticBeanstalk
    #- namespace: AWS/Lambda
    #- namespace: AWS/Redshift
    #- namespace: AWS/RDS
    #- namespace: AWS/Route53
    #- namespace: AWS/S3
    #- namespace: AWS/VPN
  access_key_id: 'key'
  secret_access_key: key’
- module: aws
  period: 5m
  metricsets:
    - dynamodb
    - ebs
    - ec2
    - lambda
    - rds
    - sns
    - sqs
    - vpn
    - usage
    - elb
  access_key_id: ‘key’
  secret_access_key: ‘key’
- module: aws
  period: 12h
  metricsets:
    - billing
  regions:
    - us-east-1
  access_key_id: ‘key’
  secret_access_key: ‘key’
- module: aws
  period: 24h
  metricsets:
    - s3_daily_storage
    - s3_request
  access_key_id: ‘key’
  secret_access_key: ‘key’

[kvch (Noémi Ványi)]

Have you run ./metribeat setup to deploy assets into Kibana and Elasticsearch?

[ozonshak]

@kvch Yes, I ran it. I had first set up metricbeat to gather some CPU statistics and such, so I believe I ran it at that time. I did a quick "history" check on Linux and I did run this a couple of times. I have since moved on to try and setup AWS and are running into these issues.

[Kaiyan_Sheng]

Hi! For only seeing 5 EC2 instances, do you mean from the EC2 dashboard? That's because in the dashboard, for better view, we have a limit on how many instances to show (top 5). You can change this limit under each visualization.

For config like this:

- module: aws
  period: 5m
  metricsets:
    - cloudwatch
  metrics:
    - namespace: "*"

You will be collecting all metrics from CloudWatch, which might introduce a large amount of AWS API cost. Just want to make sure you are aware of this

Regarding to no statistics for EBS/RDS, do you see metrics from EBS/RDS in Kibana Discover?

[ozonshak]

Thanks @Kaiyan_Sheng. Ok, so I found the visualization limit for EC2 and changed that from 5 to 20. So, I'm good there.

For RDS, I noticed in the visualization that the default index pattern was something other than metricbeat for some reason. So, I changed that to MetricBeat:

Next, I picked one of the visualizations - just chose the "database connections" one. Here I can see the correct index now, but there was no field selected. So, I chose aws.rds.database_connections. That seems to be populating that visualiaztion with data now, however, this is going to be a bit painful to try and choose the appropriate fields for all the visualizations. Any idea why these are not set and what they should be for the various visualizations?