AWS vpcflow errors - count not find region configuration, context deadline exceeded

swisscheese · March 27, 2020, 11:28pm

I have vpc flow logs going to an S3 bucket and an SQS notification for any object creation event.

I actually much preferred the logstash method of polling the bucket, for several reasons - mainly the ability to re-index from source easily, use regex matching of file patterns, and general simplicity. The realtime notifications don't provide me with much benefit and just add complexity. I hope to see polling as an option on filebeat some day.

Nonetheless, I've set it up, and I am using an instance role on my filebeat node for access to SQS and S3.

Here is my aws.yml:

module: aws
vpcflow:
enabled: true
var.queue_url: https://sqs.us-west-2.amazonaws.com//vpcflow

Although quite a bit of data is being read successfully and I can see it in Kibana, I'm getting a lot of errors and warnings. I have no easy way to tell if it's getting everything, but these messages imply that some may be failing and getting dropped. I can't be certain though, because they're very confusing messages.

2020-03-27T01:27:10.019Z ERROR [s3] s3/input.go:204 failed to receive message from SQS: MissingRegion: could not find region configuration
2020-03-27T01:27:20.019Z ERROR [s3] s3/input.go:204 failed to receive message from SQS: MissingRegion: could not find region configuration
2020-03-27T01:27:20.019Z ERROR [s3] s3/input.go:204 failed to receive message from SQS: MissingRegion: could not find region configuration
2020-03-27T01:27:30.020Z ERROR [s3] s3/input.go:204 failed to receive message from SQS: MissingRegion: could not find region configuration
2020-03-27T01:27:30.020Z ERROR [s3] s3/input.go:204 failed to receive message from SQS: MissingRegion: could not find region configuration
2020-03-27T01:27:40.020Z ERROR [s3] s3/input.go:204 failed to receive message from SQS: MissingRegion: could not find region configuration
2020-03-27T01:27:40.020Z ERROR [s3] s3/input.go:204 failed to receive message from SQS: MissingRegion: could not find region configuration
2020-03-27T01:27:40.887Z ERROR [s3] s3/input.go:475 ReadString failed: context deadline exceeded
2020-03-27T01:27:41.531Z ERROR [s3] s3/input.go:475 ReadString failed: context deadline exceeded
2020-03-27T01:27:41.531Z ERROR [s3] s3/input.go:386 createEventsFromS3Info failed for AWSLogs//vpcflowlogs/us-west-2/2020/03/26/_vpcflowlogs_us-west-2_fl-.log.gz: ReadString failed: conte
xt deadline exceeded
2020-03-27T01:27:41.571Z WARN [s3] s3/input.go:277 Processing message failed, updating visibility timeout
2020-03-27T01:27:41.636Z INFO [s3] s3/input.go:282 Message visibility timeout updated to 300
2020-03-27T01:27:41.794Z WARN [s3] s3/input.go:277 Processing message failed, updating visibility timeout
2020-03-27T01:27:41.799Z INFO [s3] s3/input.go:282 Message visibility timeout updated to 300
2020-03-27T01:27:41.993Z WARN [s3] s3/input.go:277 Processing message failed, updating visibility timeout
2020-03-27T01:27:41.997Z INFO [s3] s3/input.go:282 Message visibility timeout updated to 300

Any ideas?

Kaiyan_Sheng · March 30, 2020, 1:54pm

Hi @swisscheese, thank you for letting us know about your preference on polling method! I will create a github issue to track this.

For the error message you see in the log, I believe it's caused by other filesets that are enabled(by default) in aws module. We do have a github issue to fix this: https://github.com/elastic/beats/issues/17256

If you change aws.yml to the config below(assume you are running 7.6 version), you should see a better/cleaner log.

- module: aws
  cloudtrail:
    enabled: false
  elb:
    enabled: false
  s3access:
    enabled: false
  vpcflow:
    enabled: true
    var.queue_url: https://sqs.us-west-2.amazonaws.com//vpcflow

Kaiyan_Sheng · April 2, 2020, 11:18pm

Ahh sorry I missed this error message. Question: seems like this error showed up twice for the same message AWSLogs/699536110035/vpcflowlogs/us-west-2/2020/03/29/_vpcflowlogs_us-west-2_fl-.log.gz in 1 second, which is not cool... Did you by any chance changed the visibility_timeout param?

2020-03-30T18:30:44.613Z	ERROR	[s3]	s3/input.go:386	createEventsFromS3Info failed for AWSLogs/699536110035/vpcflowlogs/us-west-2/2020/03/29/_vpcflowlogs_us-west-2_fl-.log.gz: ReadString failed: context deadline exceeded
2020-03-30T18:30:45.207Z	ERROR	[s3]	s3/input.go:386	createEventsFromS3Info failed for AWSLogs/699536110035/vpcflowlogs/us-west-2/2020/03/29/_vpcflowlogs_us-west-2_fl-.log.gz: ReadString failed: context deadline exceeded

Do you know if this log file eventually get read by Filebeat? Or it always show in the error message repeatedly?

Kaiyan_Sheng · April 2, 2020, 11:19pm

If you could attach the actual file here, that would be great

Kaiyan_Sheng · April 2, 2020, 11:34pm

Since it's a context deadline exceeded error, could you try increase setting var.api_timeout in aws.yml please?

tbuatois · April 9, 2020, 5:48pm

Hi,

I am facing exactly the same problem as above.

My configuration :

- module: aws
  cloudtrail:
    enabled: false
  elb:
    enabled: false
  s3access:
    enabled: false
  vpcflow:
    enabled: true
    # AWS SQS queue url
    var.queue_url: https://sqs.eu-west-3.amazonaws.com/$AWSACCOUNID/$QUEUEName

I did not change any other values.

My error log :

Apr 09 19:35:21 ip-10-133-50-250.******.fr filebeat[30039]: 2020-04-09T19:35:21.769+0200        ERROR        [s3]        s3/input.go:475        ReadString failed: context deadline exceeded
Apr 09 19:35:21 ip-10-133-50-250.******.fr filebeat[30039]: 2020-04-09T19:35:21.769+0200        ERROR        [s3]        s3/input.go:386        createEventsFromS3Info failed for AWSLogs/******/vpcflowlogs/eu-west-3/2020/04/09*******_vpcflowlogs_eu-west-3_fl-0629b5eccc8f3d0ad_20200409T0120Z_9b923cb5.log.gz: ReadString failed: context deadline exceeded
Apr 09 19:35:43 ip-10-133-50-250.******.fr filebeat[30039]: 2020-04-09T19:35:43.317+0200        WARN        [s3]        s3/input.go:277        Processing message failed, updating visibility timeout
Apr 09 19:35:43 ip-10-133-50-250.******.fr filebeat[30039]: 2020-04-09T19:35:43.357+0200        INFO        [s3]        s3/input.go:282        Message visibility timeout updated to 300

My SQS queue settings :

VPCFlowLogsQueue:
    Type: AWS::SQS::Queue
    Properties:
      DelaySeconds: 0
      MaximumMessageSize: 262144
      MessageRetentionPeriod: 345600 
      VisibilityTimeout: 30

SQS queue policy sqs:* for logstash instance role.

Like @swisscheese I have a vpc flow logs data showing up but I cannot be certain that is all of it.

Elasticsearch Version : 7.5.1
Filebeat Version : 7.6.2 (Use default template/dashboard and index)

S3 and SQS Access trough instance profile role on AWS EC2. (s3:* and sqs:* on specified resources).

Any thoughts ?

Regards,
Thomas.

Kaiyan_Sheng · April 13, 2020, 9:18pm

Could you give this config a try with increasing var.api_timeout please?

- module: aws
  cloudtrail:
    enabled: false
  elb:
    enabled: false
  s3access:
    enabled: false
  vpcflow:
    enabled: true
    var.queue_url: https://sqs.eu-west-3.amazonaws.com/$AWSACCOUNID/$QUEUEName
    var.api_timeout: 720s
    var.visibility_timeout: 300s

Also since the error message is complaining about file AWSLogs/******/vpcflowlogs/eu-west-3/2020/04/09*******_vpcflowlogs_eu-west-3_fl-0629b5eccc8f3d0ad_20200409T0120Z_9b923cb5.log.gz, if you can check in Kibana discover (maybe with a filter) to see if there are events from this file please?

Thanks,
Kaiyan

system · May 11, 2020, 9:18pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.