I'd just like to verify I am attempting this correctly. I am trying to start a Logstash pipeline that first downloads some large (~500MB) ZIP files via http, then unzips them, then processes that data. I am having trouble with the first part, though.
- Should I use http_poller for this?
- I see a gzip codec. Should I be using this with .ZIP, or is that not going to work?
- I get these logs when trying with http_poller:
eb 16 20:50:28 lamp-01 systemd: Started logstash.
Feb 16 20:50:42 lamp-01 logstash: Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Feb 16 20:53:00 lamp-01 logstash: java.lang.OutOfMemoryError: Java heap space
Feb 16 20:53:00 lamp-01 logstash: Dumping heap to java_pid12860.hprof ...
Feb 16 20:53:00 lamp-01 logstash: Unable to create java_pid12860.hprof: Permission denied
Any idea why this is happening? These appear only a few minutes after starting the pipeline, and I allocated 2GB of memory to Logstash.