Filebeat AWS Module problem

Elastic Stack Beats
,
1

Hi

I am trying to use the Filebeat AWS module to retrieve data from AWS.

My Elastic environment is running v7.7 and I have a single server running Kibana, Elasticsearch, Filebeat and Logstash (Logstash not relevant for this issue). Filebeat is running the Palo Alto (panw) module and this works well, receiving events over syslog. The Filebeat AWS module is not working yet.

The installation as a whole works and all other data sources are working so I don't see any issues with the configuration between Filebeat, Elasticsearch and Kibana. When I ran filebeat setup -e there were no issues reported and AWS dashboards are in Kibana - just no data.

I have stripped the configs back to bare minimum and believe everything is ok.

My aws.yml file looks like this:

- module: aws
cloudtrail:
enabled: true
var.queue_url: https://sqs.eu-west-2.amazonaws.com/1234567890/QueueName
var.credential_profile_name: fb-aws
var.shared_credential_file: /root/.aws/credentials

Other elements in this config file are configured the same so cloudwatch, ec2, elb, s3access and vpcflow all look the same, using the same SQS URL and credentials.

there is no S3 or AWS config based information in the filebeat.yml file.

When I start filebeat with debug mode enabled I can see it processes messages, I just don't know where they go - if I search the filebeat-* index in Kibana I can't find anything relating to the AWS module. I also can't see any errors in the Elasticsearch log file to suggest a problem with processing the messages.

I have tried to extract the pertinent filebeat log data below. Initial messages show start up which I think is ok:

May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.500+0100 DEBUG [cfgfile] cfgfile/reload.go:205 Scan for new config files
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.500+0100 DEBUG [cfgfile] cfgfile/cfgfile.go:193 Load config from file: /etc/filebeat/modules.d/aws.yml
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.501+0100 DEBUG [cfgfile] cfgfile/cfgfile.go:193 Load config from file: /etc/filebeat/modules.d/panw.yml
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.501+0100 DEBUG [cfgfile] cfgfile/reload.go:224 Number of module configs found: 2
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.501+0100 DEBUG [reload] cfgfile/list.go:62 Starting reload procedure, current runners: 0
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.501+0100 DEBUG [reload] cfgfile/list.go:80 Start list: 2, Stop list: 0

May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.543+0100 DEBUG [modules] fileset/pipelines.go:67 Required processors:
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.544+0100 DEBUG [esclientleg] eslegclient/connection.go:312 GET https://:9200/_ingest/pipeline/filebeat-7.7.0-aws-cloudtrail-pipeline
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.546+0100 DEBUG [modules] fileset/pipelines.go:120 Pipeline filebeat-7.7.0-aws-cloudtrail-pipeline already loaded
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.546+0100 DEBUG [modules] fileset/pipelines.go:67 Required processors:
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.546+0100 DEBUG [esclientleg] eslegclient/connection.go:312 GET https://:9200/_ingest/pipeline/filebeat-7.7.0-aws-cloudwatch-pipeline
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.547+0100 DEBUG [modules] fileset/pipelines.go:120 Pipeline filebeat-7.7.0-aws-cloudwatch-pipeline already loaded
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.547+0100 DEBUG [modules] fileset/pipelines.go:67 Required processors:
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.548+0100 DEBUG [esclientleg] eslegclient/connection.go:312 GET https://:9200/_ingest/pipeline/filebeat-7.7.0-aws-ec2-pipeline
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.548+0100 DEBUG [modules] fileset/pipelines.go:120 Pipeline filebeat-7.7.0-aws-ec2-pipeline already loaded
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.548+0100 DEBUG [modules] fileset/pipelines.go:67 Required processors: [{geoip ingest-geoip}]
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.548+0100 DEBUG [esclientleg] eslegclient/connection.go:312 GET https://:9200/_nodes/ingest
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.551+0100 DEBUG [esclientleg] eslegclient/connection.go:312 GET https://:9200/_ingest/pipeline/filebeat-7.7.0-aws-elb-pipeline
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.552+0100 DEBUG [modules] fileset/pipelines.go:120 Pipeline filebeat-7.7.0-aws-elb-pipeline already loaded
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.552+0100 DEBUG [modules] fileset/pipelines.go:67 Required processors:
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.554+0100 DEBUG [esclientleg] eslegclient/connection.go:312 GET https://:9200/_ingest/pipeline/filebeat-7.7.0-aws-s3access-pipeline
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.555+0100 DEBUG [modules] fileset/pipelines.go:120 Pipeline filebeat-7.7.0-aws-s3access-pipeline already loaded
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.555+0100 DEBUG [modules] fileset/pipelines.go:67 Required processors:
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.556+0100 DEBUG [esclientleg] eslegclient/connection.go:312 GET https://:9200/_ingest/pipeline/filebeat-7.7.0-aws-vpcflow-pipeline
May 18 16:12:47 server filebeat[17797]: 2020-05-18T16:12:47.557+0100 DEBUG [modules] fileset/pipelines.go:120 Pipeline filebeat-7.7.0-aws-vpcflow-pipeline already loaded

The logs show a message being processed

May 18 16:13:14 server filebeat[17797]: 2020-05-18T16:13:14.415+0100 DEBUG [s3] s3/input.go:238 Processing 1 messages
May 18 16:13:14 server filebeat[17797]: 2020-05-18T16:13:14.415+0100 DEBUG [s3] s3/input.go:258 handleSQSMessage succeed and returned 0 sets of S3 log info
May 18 16:13:14 server filebeat[17797]: 2020-05-18T16:13:14.415+0100 DEBUG [s3] s3/input.go:267 handleS3Objects succeed
May 18 16:13:14 server filebeat[17797]: 2020-05-18T16:13:14.415+0100 DEBUG [s3] s3/input.go:288 Deleting message from SQS: 0xc00060aed0

This is a subsequent, shows successful connection but no messages to process

May 18 16:13:44 server filebeat[17797]: 2020-05-18T16:13:44.451+0100 DEBUG [s3] s3/input.go:211 no message received from SQS:https://sqs.eu-west-2.amazonaws.com/123456789/QueueName

I'm not sure what additional debug I can do to track where the problem is?

Any help appreciated.

2

Hi! Thanks for posting your question here. In AWS SQS portal, do you see message number > 1 ? Seem like Filebeat does not see any message from the queue specified in the config.

3

Hi

Thank you for responding!

I just stopped filebeat for a few minutes and monitored the SQS queue in the AWS portal, this built up to 19 messages available. I then restarted filebeat and the queue cleared. Monitoring the filebeat log I can also see that during start-up it processed bigger numbers - it is still a bit of a test AWS environment so not overly busy. If I also look at the monitoring tab on the sQS page in AWS I can see the NumberOfMessagesSent chart shows a regular stream of events with the line between 30-50 events.

Thanks

Phil

4

I have now got it working. This blog was useful

Elastic Blog

The main article was slightly different from my use case but the AWS configuration helped. Our SQS was receiving notifications of the new log event from the SNS so I think Filebeat knew there was a log entry to collect, just didn't know where to go get it. Step 3 in this blog shows the S3 settings to set a notification and this got it working.

It may be useful if the Elastic documentation gave more detailed AWS set up (unless it is there and I couldn't find it).

1 Like
5

Great!! Thank you for the feedback!! I will get the documentation updated soon for sure.

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.