# How to process multiple files in logstash **URL:** \ **Category:** Logstash\ **Created:** [August 15, 2017, 8:34pm UTC](https://discuss.elastic.co/t/how-to-process-multiple-files-in-logstash/97148 "2017-08-15T20:34:20Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Post date:** [August 15, 2017, 8:34pm UTC](https://discuss.elastic.co/t/how-to-process-multiple-files-in-logstash/97148/1 "2017-08-15T20:34:20Z")
I have some million json files that i separated in multiple sub folders. Basically these are Cloudtrail data for a year. My Logstash version is 5.5 and this is a 8 core , 32 GB system. The problem is that when i run my logstash that uses a file input plugin and outputting to elasticsearch. It runs for couple of time and then dies with java heap space. Can someone please help me on this. I am running out of ideas now. -- Niraj ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 16, 2017, 7:09pm UTC](https://discuss.elastic.co/t/how-to-process-multiple-files-in-logstash/97148/2 "2017-08-16T19:09:59Z")
The file input isn't built to process filename patterns that expand to millions of files. You'll have to process them in smaller numbers, e.g. by writing a small script that reads the millions of files and copies the data to a small(er) set of files that you point Logstash to. Another option could be to send the file to Logstash over a socket or a broker. A broker like RabbitMQ will help you with backpressure if Logstash isn't able to consume the messages fast enough. ---
**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Post date:** [August 16, 2017, 8:35pm UTC](https://discuss.elastic.co/t/how-to-process-multiple-files-in-logstash/97148/3 "2017-08-16T20:35:40Z")
Thanks @magnusbaeck. Can you recommend a solution where i have \*.gz files coming in from amazon cloudtrail and those have json files in it and these json files doesn't have a new line in it. Is there a way i can ingest files without having to process of unpacking the zip and adding new line to every json present. I am having a hard time processing these data. -- Niraj ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 17, 2017, 5:09am UTC](https://discuss.elastic.co/t/how-to-process-multiple-files-in-logstash/97148/4 "2017-08-17T05:09:43Z")
If these files are among the million files you'll probably have to process them outside of Logstash anyway so I don't know if it's such a big problem. Not sure what you mean by "doesn't have a new line in it". Are the files lacking a trailing newline or what? ---
**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Post date:** [August 17, 2017, 7:41am UTC](https://discuss.elastic.co/t/how-to-process-multiple-files-in-logstash/97148/5 "2017-08-17T07:41:29Z")
@magnusbaeck Yes the json files doesn't have an EOL in it. I have to use echo \>\> filename.json to add an end of line to it and it works after that. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 14, 2017, 7:41am UTC](https://discuss.elastic.co/t/how-to-process-multiple-files-in-logstash/97148/6 "2017-09-14T07:41:37Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.