metricbeat version 7.10.0 (amd64), libbeat 7.10.0
elasticsearch version: 7.10.0
kibana version: 7.10.0
Good day,
I'm still trying to collect and display the AWS metrics, especially EC2-metrics and billing information with the Metricbeat AWS module. I'm using the metricsets: cloudwatch (namespace: AWS/EC2) and billing.
Metricbeat is installed on one of the AWS EC2 instances.
Currently Elasticsearch sucessfully receives data from metricbeat, which I am very happy about.
So I have a non-empty and growing metricbeat-7.10.0-2021.01.11-000001-index.
But all the default AWS Dashboards in Kibana are expecting data fields that my index doesn't have.
These are the the available fields in the metricbeat-* Index pattern (Kibana Discover):
_id
_index
_score
_type
@timestamp
agent.ephemral_id
agent.hostname
agent.id
agent.name
agent.type
agent.version
cloud.account.id
cloud.availability_zone
cloud.image.id
cloud.instance.id
cloud.machin.type
cloud.provides
cloud.region
ecs.version
event.dataset
event.duration
event.module
host.cpu.pct
host.network.in.bytes
host.network.in.packets
host.network.out.bytes
host.network.out.packes
metricset.name
metricset.period
process.args
process.executable
process.name
process.pgid
process.pid
process.ppid
process.working_directory
service.tpye
system.cpu.cores
system.cpu...
system.filesystem.available
system.filesystem...
system.load.1
system.load...
system.memory.free
system.memory...
system.network..
system.process...
system.socket...
user.name
The [Metricbeat AWS] Billing Overview expects fields like aws.billing....
The [Metricbeat AWS] Overview and the EC2 Overview dashboards expect fields in the form of aws.ec2.
which are all missing from my index.
This is my metricbeat.yml:
path.home: /usr/share/metricbeat
path.config: /etc/metricbeat
path.data: /var/lib/metricbeat
path.logs: /var/log/
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
processors:
- add_cloud_metadata: ~
- add_docker_metadata: ~
output.elasticsearch:
hosts: ["https://localhost:9200"]
username: elastic
password: <pw>
ssl.certificate_authorities: ["<path-to>-ca.crt"]
setup.kibana:
host: "http://localhost:5601"
username: elastic
password: <pw>
my aws.yml:
- module: aws
period: 300s
metricsets:
- cloudwatch
metrics:
- namespace: AWS/EC2
resource_type: ec2:instance
credential_profile_name: <profile-name>
access_key_id: '<access-key>'
secret_access_key: '<secret-key>'
- module: aws
period: 24h
metricsets:
- billing
access_key_id: '<access-key>'
secret_access_key: '<secret-key>'
cost_explorer_config:
group_by_dimension_keys:
- "AZ"
- "INSTANCE_TYPE"
- "SERVICE"
group_by_tag_keys:
- "aws:createdBy"
I would appreciate any help.