Logstash gzip_pattern matching

Elastic Stack Logstash
1

Hi there,

I'm using ELK 7.7. I need to ingest s3 objects with description exactly like this

We are using S3 input plugin to pull data from S3, we are using kinesis firehose to push data to S3, which does not adds 'gz' extension to the files, but data is in zipped format. S3 input plugin is pulling data from S3 it can only decompress the data which has gz as extension since while code walk through we we notice there is a condition checking 'gz' as extension then only it decompress the file data.

I know of this gzip_pattern flag in logstash s3 plugin that resolves this.

How exactly do I use it to match all files (no gz or gzip extention) in my S3 folder (s3bucket/cloudwatchlogs/yyyy/mm/dd/hh/filename) which consists only of files pulled from cloudwatch?

Thanks in advance
ck

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.