I'd just like to verify I am attempting this correctly. I am trying to start a Logstash pipeline that first downloads some large (~500MB) ZIP files via http, then unzips them, then processes that data. I am having trouble with the first part, though. Should I use http_poller for this? I see a gzip…

I don't think there's a reasonable way of doing this with Logstash alone. I suggest you run a script alongside Logstash that downloads the zip files, unpacks them, and hands them over to Logstash. I see a gzip codec. Should I be using this with .ZIP, or is that not going to work? It's not going …

Processing remote ZIP files with Logstash

Elastic Stack Logstash

arisbanach February 16, 2018, 9:14pm 3

I don't have any codec currently specified though. Just a simple config with http_poller set on a cron and a file output location.

Does logstash support zip format logs from s3 bucket

Topic		Replies	Views
How to efficiently parse log files in logstash? Logstash	3	421	July 16, 2018
Http_poller input Logstash	3	251	May 26, 2020
Does Logstash (7.0.1) support gzip file input? Logstash	7	4849	June 20, 2019
Does logstash support zip format logs from s3 bucket Logstash	8	642	November 15, 2022
Logstash http_poller - fetching big files Logstash	1	365	June 25, 2019

Processing remote ZIP files with Logstash

Related topics