How to read S3 bucket using filebeat and parse in logstash

Elastic Stack Beats
1

Can some one explain how to read the contents of S3 bucket and store in Elasticsearch via logstash

2

Hi @vijay117

Do you want to use filebeat or logstash or both... which one?

A) filebeat->logstash->elasticsearch
B) filebeat-> elasticsearch
C) logstash->elasticsearch

Filebeat s3 input

Logstash s3 input

This looks like a duplicate of

3

My suggestion is to use Filebeat or Elastic Agent to get data from S3 and also use SQS queues to receive notification when a new file is created on the S3 bucket.

Logstash s3 input does not work well in medium/large buckets.

1 Like
4

Here is an article on that

5

Thanks stephen, i only wanted to list the files in a folder in bucket, some how the s3 plugin doesnot work that way, had to create a customised script to access bucket and list the files