I am trying to use the aws module for filebeat. My goal is to recreate the ELB Inbound Traffic [Filebeat AWS] visualization. It does not work by default. I've added an s3 bucket for collecting elb logs and an s3 bucket notification that is connected to an sqs queue. This queue is used as an input in the aws.yml like so
elb:
enabled: true
# AWS SQS queue url
var.queue_url: ${ELB_SQS}
When I try to add a data panel to sum the source.bytes field the selector does not show the field. Nor does the field exist when I look at the records from my ellb logs. Here is a screenshot
I would love it if someone could help me with this. I've also attached my filebeat configuration for reference too.
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.kibana.host: "host"
setup.kibana.protocol: "https"
setup.kibana.username: "user"
setup.kibana.password: password
setup.dashboards.enabled: true
setup.dashboards.directory: ${path.config}/kibana
setup.ilm.enabled: false
logging.level: debug
output.elasticsearch:
hosts: ["host"]
protocol: "https"
username: "user"
password: password
bulk_max_size: 3200
worker: 16
indices:
- index: "cloudtrail-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
event.dataset: "aws.cloudtrail"
- index: "elb-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
event.dataset: "aws.elb"
- index: "vpc-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
event.dataset: "aws.vpc"
processors:
- add_fields:
target: target
fields:
environment: ${ENV}