I'd just like to verify I am attempting this correctly. I am trying to start a Logstash pipeline that first downloads some large (~500MB) ZIP files via http, then unzips them, then processes that data. I am having trouble with the first part, though. Should I use http_poller for this? I see a gzip…

I don't think there's a reasonable way of doing this with Logstash alone. I suggest you run a script alongside Logstash that downloads the zip files, unpacks them, and hands them over to Logstash. I see a gzip codec. Should I be using this with .ZIP, or is that not going to work? It's not going …

Processing remote ZIP files with Logstash

Elastic Stack Logstash

arisbanach February 16, 2018, 9:14pm 3

I don't have any codec currently specified though. Just a simple config with http_poller set on a cron and a file output location.

Does logstash support zip format logs from s3 bucket

Topic		Replies	Views
How to efficiently parse log files in logstash? Logstash	3	421	July 16, 2018
Http_poller input Logstash	3	250	May 26, 2020
Does logstash support zip format logs from s3 bucket Logstash	8	637	November 15, 2022
java.lang.OutOfMemoryError: Java heap space (logstash, http_poller) Elasticsearch elastic-stack-monitoring	1	164	July 12, 2023
Error on parsing gzip files "Unexpected end of ZLIB input stream" Logstash	1	574	August 12, 2021

Processing remote ZIP files with Logstash

Related Topics