I'd just like to verify I am attempting this correctly. I am trying to start a Logstash pipeline that first downloads some large (~500MB) ZIP files via http, then unzips them, then processes that data. I am having trouble with the first part, though. Should I use http_poller for this? I see a gzip…

I don't think there's a reasonable way of doing this with Logstash alone. I suggest you run a script alongside Logstash that downloads the zip files, unpacks them, and hands them over to Logstash. I see a gzip codec. Should I be using this with .ZIP, or is that not going to work? It's not going …

Discuss the Elastic Stack

Processing remote ZIP files with Logstash

Elastic Stack Logstash

arisbanach February 16, 2018, 9:14pm 3

I don't have any codec currently specified though. Just a simple config with http_poller set on a cron and a file output location.

Does logstash support zip format logs from s3 bucket

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.