# Filebeat AWS Module problem
**URL:** https://discuss.elastic.co/t/filebeat-aws-module-problem/233141
**Category:** Beats
**Tags:** beats-module, filebeat
**Created:** [May 18, 2020, 3:40pm UTC](https://discuss.elastic.co/t/filebeat-aws-module-problem/233141 "2020-05-18T15:40:11Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@PhilA](https://discuss.elastic.co/u/PhilA)
#### Post date: [May 18, 2020, 3:40pm UTC](https://discuss.elastic.co/t/filebeat-aws-module-problem/233141/1 "2020-05-18T15:40:11Z")
Hi
I am trying to use the Filebeat AWS module to retrieve data from AWS.
My Elastic environment is running v7.7 and I have a single server running Kibana, Elasticsearch, Filebeat and Logstash (Logstash not relevant for this issue). Filebeat is running the Palo Alto (panw) module and this works well, receiving events over syslog. The Filebeat AWS module is not working yet.
The installation as a whole works and all other data sources are working so I don't see any issues with the configuration between Filebeat, Elasticsearch and Kibana. When I ran `filebeat setup -e` there were no issues reported and AWS dashboards are in Kibana - just no data.
I have stripped the configs back to bare minimum and believe everything is ok.
My aws.yml file looks like this:
> - module: aws
> cloudtrail:
> enabled: true
> var.queue\_url: [https://sqs.eu-west-2.amazonaws.com/1234567890/QueueName](https://sqs.eu-west-2.amazonaws.com/1234567890/QueueName)
> var.credential\_profile\_name: fb-aws
> var.shared\_credential\_file: /root/.aws/credentials
Other elements in this config file are configured the same so cloudwatch, ec2, elb, s3access and vpcflow all look the same, using the same SQS URL and credentials.
there is no S3 or AWS config based information in the filebeat.yml file.
When I start filebeat with debug mode enabled I can see it processes messages, I just don't know where they go - if I search the filebeat-\* index in Kibana I can't find anything relating to the AWS module. I also can't see any errors in the Elasticsearch log file to suggest a problem with processing the messages.
I have tried to extract the pertinent filebeat log data below. Initial messages show start up which I think is ok:
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.500+0100 DEBUG [cfgfile] cfgfile/reload.go:205 Scan for new config files
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.500+0100 DEBUG [cfgfile] cfgfile/cfgfile.go:193 Load config from file: /etc/filebeat/modules.d/aws.yml
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.501+0100 DEBUG [cfgfile] cfgfile/cfgfile.go:193 Load config from file: /etc/filebeat/modules.d/panw.yml
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.501+0100 DEBUG [cfgfile] cfgfile/reload.go:224 Number of module configs found: 2
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.501+0100 DEBUG [reload] cfgfile/list.go:62 Starting reload procedure, current runners: 0
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.501+0100 DEBUG [reload] cfgfile/list.go:80 Start list: 2, Stop list: 0
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.543+0100 DEBUG [modules] fileset/pipelines.go:67 Required processors:
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.544+0100 DEBUG [esclientleg] eslegclient/connection.go:312 GET https://:9200/\_ingest/pipeline/filebeat-7.7.0-aws-cloudtrail-pipeline
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.546+0100 DEBUG [modules] fileset/pipelines.go:120 Pipeline filebeat-7.7.0-aws-cloudtrail-pipeline already loaded
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.546+0100 DEBUG [modules] fileset/pipelines.go:67 Required processors:
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.546+0100 DEBUG [esclientleg] eslegclient/connection.go:312 GET https://:9200/\_ingest/pipeline/filebeat-7.7.0-aws-cloudwatch-pipeline
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.547+0100 DEBUG [modules] fileset/pipelines.go:120 Pipeline filebeat-7.7.0-aws-cloudwatch-pipeline already loaded
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.547+0100 DEBUG [modules] fileset/pipelines.go:67 Required processors:
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.548+0100 DEBUG [esclientleg] eslegclient/connection.go:312 GET https://:9200/\_ingest/pipeline/filebeat-7.7.0-aws-ec2-pipeline
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.548+0100 DEBUG [modules] fileset/pipelines.go:120 Pipeline filebeat-7.7.0-aws-ec2-pipeline already loaded
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.548+0100 DEBUG [modules] fileset/pipelines.go:67 Required processors: [{geoip ingest-geoip}]
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.548+0100 DEBUG [esclientleg] eslegclient/connection.go:312 GET https://:9200/\_nodes/ingest
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.551+0100 DEBUG [esclientleg] eslegclient/connection.go:312 GET https://:9200/\_ingest/pipeline/filebeat-7.7.0-aws-elb-pipeline
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.552+0100 DEBUG [modules] fileset/pipelines.go:120 Pipeline filebeat-7.7.0-aws-elb-pipeline already loaded
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.552+0100 DEBUG [modules] fileset/pipelines.go:67 Required processors:
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.554+0100 DEBUG [esclientleg] eslegclient/connection.go:312 GET https://:9200/\_ingest/pipeline/filebeat-7.7.0-aws-s3access-pipeline
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.555+0100 DEBUG [modules] fileset/pipelines.go:120 Pipeline filebeat-7.7.0-aws-s3access-pipeline already loaded
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.555+0100 DEBUG [modules] fileset/pipelines.go:67 Required processors:
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.556+0100 DEBUG [esclientleg] eslegclient/connection.go:312 GET https://:9200/\_ingest/pipeline/filebeat-7.7.0-aws-vpcflow-pipeline
> May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.557+0100 DEBUG [modules] fileset/pipelines.go:120 Pipeline filebeat-7.7.0-aws-vpcflow-pipeline already loaded
The logs show a message being processed
> May 18 16:13:14 server filebeat[17797]: 2020-05-18T16:13:14.415+0100 DEBUG [s3] s3/input.go:238 Processing 1 messages
> May 18 16:13:14 server filebeat[17797]: 2020-05-18T16:13:14.415+0100 DEBUG [s3] s3/input.go:258 handleSQSMessage succeed and returned 0 sets of S3 log info
> May 18 16:13:14 server filebeat[17797]: 2020-05-18T16:13:14.415+0100 DEBUG [s3] s3/input.go:267 handleS3Objects succeed
> May 18 16:13:14 server filebeat[17797]: 2020-05-18T16:13:14.415+0100 DEBUG [s3] s3/input.go:288 Deleting message from SQS: 0xc00060aed0
This is a subsequent, shows successful connection but no messages to process
> May 18 16:13:44 server filebeat[17797]: 2020-05-18T16:13:44.451+0100 DEBUG [s3] s3/input.go:211 no message received from SQS:[https://sqs.eu-west-2.amazonaws.com/123456789/QueueName](https://sqs.eu-west-2.amazonaws.com/123456789/QueueName)
I'm not sure what additional debug I can do to track where the problem is?
Any help appreciated.
---
### Author:  [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)
#### Post date: [May 19, 2020, 3:03am UTC](https://discuss.elastic.co/t/filebeat-aws-module-problem/233141/2 "2020-05-19T03:03:30Z")
Hi! Thanks for posting your question here. In AWS SQS portal, do you see message number \> 1 ? Seem like Filebeat does not see any message from the queue specified in the config.
---
### Author:  [@PhilA](https://discuss.elastic.co/u/PhilA)
#### Post date: [May 19, 2020, 5:10am UTC](https://discuss.elastic.co/t/filebeat-aws-module-problem/233141/3 "2020-05-19T05:10:58Z")
> [@Kaiyan\_Sheng](#):
>
> S SQS portal, do you see messag
Hi
Thank you for responding!
I just stopped filebeat for a few minutes and monitored the SQS queue in the AWS portal, this built up to 19 messages available. I then restarted filebeat and the queue cleared. Monitoring the filebeat log I can also see that during start-up it processed bigger numbers - it is still a bit of a test AWS environment so not overly busy. If I also look at the monitoring tab on the sQS page in AWS I can see the NumberOfMessagesSent chart shows a regular stream of events with the line between 30-50 events.
Thanks
Phil
---
### Author:  [@PhilA](https://discuss.elastic.co/u/PhilA)
#### Post date: [May 19, 2020, 2:46pm UTC](https://discuss.elastic.co/t/filebeat-aws-module-problem/233141/4 "2020-05-19T14:46:17Z")
I have now got it working. This blog was useful
[Elastic Blog](https://www.elastic.co/blog/getting-aws-logs-from-s3-using-filebeat-and-the-elastic-stack)
The main article was slightly different from my use case but the AWS configuration helped. Our SQS was receiving notifications of the new log event from the SNS so I think Filebeat knew there was a log entry to collect, just didn't know where to go get it. Step 3 in this blog shows the S3 settings to set a notification and this got it working.
It may be useful if the Elastic documentation gave more detailed AWS set up (unless it is there and I couldn't find it).
---
### Author:  [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)
#### Post date: [May 19, 2020, 4:10pm UTC](https://discuss.elastic.co/t/filebeat-aws-module-problem/233141/5 "2020-05-19T16:10:14Z")
Great!! Thank you for the feedback!! I will get the documentation updated soon for sure.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 16, 2020, 4:10pm UTC](https://discuss.elastic.co/t/filebeat-aws-module-problem/233141/6 "2020-06-16T16:10:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.