# Processing remote ZIP files with Logstash **URL:** \ **Category:** Logstash\ **Created:** [February 16, 2018, 8:57pm UTC](https://discuss.elastic.co/t/processing-remote-zip-files-with-logstash/120254 "2018-02-16T20:57:23Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![arisbanach](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arisbanach](https://discuss.elastic.co/u/arisbanach)\ **Post date:** [February 16, 2018, 8:57pm UTC](https://discuss.elastic.co/t/processing-remote-zip-files-with-logstash/120254/1 "2018-02-16T20:57:23Z")
I'd just like to verify I am attempting this correctly. I am trying to start a Logstash pipeline that first downloads some large (~500MB) ZIP files via http, then unzips them, then processes that data. I am having trouble with the first part, though. 1. Should I use http\_poller for this? 2. I see a gzip codec. Should I be using this with .ZIP, or is that not going to work? 3. I get these logs when trying with http\_poller: > eb 16 20:50:28 lamp-01 systemd[1]: Started logstash. > Feb 16 20:50:42 lamp-01 logstash[12860]: Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties > Feb 16 20:53:00 lamp-01 logstash[12860]: java.lang.OutOfMemoryError: Java heap space > Feb 16 20:53:00 lamp-01 logstash[12860]: Dumping heap to java\_pid12860.hprof ... > Feb 16 20:53:00 lamp-01 logstash[12860]: Unable to create java\_pid12860.hprof: Permission denied Any idea why this is happening? These appear only a few minutes after starting the pipeline, and I allocated 2GB of memory to Logstash. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 16, 2018, 9:12pm UTC](https://discuss.elastic.co/t/processing-remote-zip-files-with-logstash/120254/2 "2018-02-16T21:12:57Z")
If I read it correctly, a Zlib::GzipReader reads the whole object (500 MB) into memory and decompresses it. I am not surprised it blows up a 2 GB heap. To me it does not appear to be a stream reader. I would experiment with much smaller files and see what the impact on the verbose GC logs looks like. But then I am a GC nerd. ---
**Author:** ![arisbanach](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arisbanach](https://discuss.elastic.co/u/arisbanach)\ **Post date:** [February 16, 2018, 9:14pm UTC](https://discuss.elastic.co/t/processing-remote-zip-files-with-logstash/120254/3 "2018-02-16T21:14:44Z")
I don't have any codec currently specified though. Just a simple config with http\_poller set on a cron and a file output location. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [February 19, 2018, 7:37am UTC](https://discuss.elastic.co/t/processing-remote-zip-files-with-logstash/120254/4 "2018-02-19T07:37:10Z")
I don't think there's a reasonable way of doing this with Logstash alone. I suggest you run a script alongside Logstash that downloads the zip files, unpacks them, and hands them over to Logstash. > I see a gzip codec. Should I be using this with .ZIP, or is that not going to work? It's not going to work. Gzip just compresses a stream of data, zip is an archive file format that also compresses. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 19, 2018, 7:37am UTC](https://discuss.elastic.co/t/processing-remote-zip-files-with-logstash/120254/5 "2018-03-19T07:37:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.