Filebeat S3 Input - Output Garbled

Elastic Stack Beats
1

I have configured VPC Flow logs to ship to S3 and then an SQS message queue to notify Filebeat.
This creates records in ES but they appear like this! Any ideas?

s3-to-es
s3-to-es.png1248×619 76.7 KB

2

Hi @jbws, thanks for giving this a try! What version of Filebeat are you using?

3

Version is 7.4.2 running in a container.

Config:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.inputs:
  - type: s3
    queue_url: zzz
    access_key_id: aaa
    secret_access_key: bbb
    visibility_timeout: 1200

cloud.id: yyy
cloud.auth: xxx

output.elasticsearch:

setup.ilm.enabled: false

Does the s3 input plugin automatically unzip .gz files?

Thanks

4

Thanks for the info! Yeah it's because VPC logs are gz files. https://github.com/elastic/beats/pull/13980 added the support and will be released in 7.5 Filebeat.

5

Also I just want to point out, we have a ticket to add a VPC fileset in Filebeat as well: https://github.com/elastic/beats/issues/13880

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.