# How to process large data with logstash

**URL:** <https://discuss.elastic.co/t/how-to-process-large-data-with-logstash/96663>\
**Category:** Logstash\
**Created:** [August 10, 2017, 7:31pm UTC](https://discuss.elastic.co/t/how-to-process-large-data-with-logstash/96663 "2017-08-10T19:31:38Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![niraj\_kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/niraj_kumar/32/4130_2.png) [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\
**Post date:** [August 10, 2017, 7:31pm UTC](https://discuss.elastic.co/t/how-to-process-large-data-with-logstash/96663/1 "2017-08-10T19:31:38Z")

</div>

I am having a trouble making my logstash work with around million files in it.

I am seeing a max\_open\_file limit warning when running it.

```
[2017-08-10T12:26:24,844][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<Java::JavaNet::URI:0x305f9956>]}
[2017-08-10T12:26:24,947][INFO][logstash.pipeline] Starting pipeline {"id"=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>1000}
[2017-08-10T12:26:25,112][INFO][logstash.pipeline] Pipeline main started
[2017-08-10T12:26:25,152][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
[2017-08-10T12:28:24,112][WARN][logstash.inputs.file] Reached open files limit: 4096, set by the 'max_open_files' option or default, files yet to open: 1784323
[2017-08-10T12:28:46,289][WARN][logstash.inputs.file] Reached open files limit: 4096, set by the 'max_open_files' option or default, files yet to open: 1784323
[2017-08-10T12:29:59,317][WARN][logstash.inputs.file] Reached open files limit: 4096, set by the 'max_open_files' option or default, files yet to open: 1784323
[2017-08-10T12:30:20,848][WARN][logstash.inputs.file] Reached open files limit: 4096, set by the 'max_open_files' option or default, files yet to open: 1784323
[2017-08-10T12:30:42,944][WARN][logstash.inputs.file] Reached open files limit: 4096, set by the 'max_open_files' option or default, files yet to open: 1784323

```

Any suggestion would help. I am not sure how to proceed with this.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 7, 2017, 7:31pm UTC](https://discuss.elastic.co/t/how-to-process-large-data-with-logstash/96663/2 "2017-09-07T19:31:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.