# Filebeat AWS module not parsing all log files

**URL:** <https://discuss.elastic.co/t/filebeat-aws-module-not-parsing-all-log-files/233976>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 23, 2020, 4:46am UTC](https://discuss.elastic.co/t/filebeat-aws-module-not-parsing-all-log-files/233976 "2020-05-23T04:46:52Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![bivaswap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bivaswap/32/47121_2.png) [@bivaswap](https://discuss.elastic.co/u/bivaswap)\
**Post date:** [May 23, 2020, 4:46am UTC](https://discuss.elastic.co/t/filebeat-aws-module-not-parsing-all-log-files/233976/1 "2020-05-23T04:46:52Z")

</div>

Hello,

I am trying to parse AWS vpcflowlogs.

After configuring AWS module with vpcflow fileset, logs started flowing.

With my current traffic condition, AWS generating around 6 million lines of log per 6 hours.  
Each logfile/s3 object contain around 80-90 thousand lines of log.

But I noticed a weird thing.

2.7 - 3.0 million logs has been parsed correctly.  
Rest of the logs didn't parsed.  
but I can see file content in message field.

More specifically, filebeat is not parsing all log files/s3 object.  
If filebeat fails to parse a file, it is not parsing a single line from that log file.

Filebeat Version 7.7  
There is error/warning message on filebeat log files.

filebeat.yml

```auto
filebeat.inputs:
- type: s3
  enabled: true
  queue_url: https://sqs.us-east-1.amazonaws.com/xxxxxxxxxxxx/vpcflowlogs-s3-notifocation

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

logging:
  metrics.enabled: false

output.elasticsearch:
  hosts: ["es-node-01.xyz.local:9200", "es-node-02.xyz.local:9200", "es-node-03.xyz.local:9200"]

```

aws.yml

```auto
- module: aws
  cloudtrail:
    enabled: false
  cloudwatch:
    enabled: false
  ec2:
    enabled: false
  elb:
    enabled: false
  s3access:
    enabled: false
  vpcflow:
    enabled: true
    var.queue_url: https://sqs.us-east-1.amazonaws.com/xxxxxxxxxx/vpcflowlogs-s3-notifocation
    var.visibility_timeout: 900s

```

Expecting to parse all log files.  
Can you please help me with that ?

---

<div class="post-metadata">

**Author:** ![zero.lim](https://avatars.discourse-cdn.com/v4/letter/z/bc79bd/32.png) [@zero.lim](https://discuss.elastic.co/u/zero.lim)\
**Post date:** [May 29, 2020, 7:00am UTC](https://discuss.elastic.co/t/filebeat-aws-module-not-parsing-all-log-files/233976/2 "2020-05-29T07:00:42Z")

</div>

Hi All,

Same problem as i happened .

When my filebeat processes a large amount of data in a short period of time, he will not be able to cut the field and put all the required data in the message field!

Currently I open the virtual machine to test filebeat!  
when data more then 1000hits per / min field was crashed  
when data less then 1000hits per / min field will be processed

Sincerely  
Zero

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 26, 2020, 7:00am UTC](https://discuss.elastic.co/t/filebeat-aws-module-not-parsing-all-log-files/233976/3 "2020-06-26T07:00:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.