Metricbeat 7.4.2-1limiting the data collection of EC-2 instances

[mylux (Luan Andrade)]

Hello everybody,
I am having issues with Metricbeat using the aws module and ec2 metricset.
I have started the service waited for something about 40 minutes and seen that only few instances had their data collected as shown below.

My configuration looks like this:

/etc/metricbeat/modules.d/aws.yml

- module: aws
period: 300s
metricsets:
 - ec2
access_key_id: '*****'
secret_access_key: '******'
regions:
 - sa-east-1
 - us-east-1

/etc/metricbeat/metricbeat.yml

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
setup.dashboards.enabled: false
setup.kibana:
  host: "mykibanahost:5601"
output.elasticsearch:
  hosts: ["http://myeshost:9200"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/metricbeat
  name: metricbeat
  keepfiles: 7
  permissions: 0644

And here you can see by this screenshot that only 31 instances

And 222 instances in my AWS console

Can you please help me finding out what I can be doing wrong?

Thanks in advance

[Kaiyan_Sheng]

Hi, thanks for posting your question here! Do you know where are these 31 instances located? From the same region or different?
Your metricbeat config looks good to me but could you try separating the regions into two config sessions just to try it out please? Not sure if that will show any difference

- module: aws
  period: 300s
  metricsets:
    - ec2
  access_key_id: '*****'
  secret_access_key: '******'
  regions:
    - sa-east-1
- module: aws
  period: 300s
  metricsets:
    - ec2
  access_key_id: '*****'
  secret_access_key: '******'
  regions:
    - us-east-1

Thanks!