# Logstash to process multiple files of large size?

**URL:** <https://discuss.elastic.co/t/logstash-to-process-multiple-files-of-large-size/300299>\
**Category:** Logstash\
**Created:** [March 22, 2022, 8:43am UTC](https://discuss.elastic.co/t/logstash-to-process-multiple-files-of-large-size/300299 "2022-03-22T08:43:14Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shreesh\_Narayanan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shreesh_narayanan/32/87312_2.png) [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)\
**Post date:** [March 22, 2022, 8:43am UTC](https://discuss.elastic.co/t/logstash-to-process-multiple-files-of-large-size/300299/1 "2022-03-22T08:43:14Z")

</div>

Hello,

To give a little bit of context, i have 3 VMs , each of 16 CPU cores out of which 2 are of 16GB RAM (they host Elasticsearch and kibana + a flask app, that provides a front end for the user to input the machineip to fetch logs )

The VM with logstash has 24GB of RAM . these Logs are of variable sizes , sometimes as large as 15GB and some as small as 150MB/1GB . my question is logstash seems to process one log at a time even though there are multiple files

Here;s lsof output that shows logstash processing 1 file, even though there are 2 .

```auto
root@dev-elk-app02:/app/input# lsof *
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 7218 root 118r REG 253,0 38579356903 131074 22.223.11.5_B-V-43YX5ZH_ddc.log
java 7218 root 119r REG 253,0 38579356903 131074 22.223.11.5_B-V-43YX5ZH_ddc.log
java 7218 root 120r REG 253,0 38579356903 131074 22.223.11.5_B-V-43YX5ZH_ddc.log
root@dev-elk-app02:/app/input# ls
189.21.70.65_ddc.log 22.223.11.5_ddc.log 20.221.11.46_ghostddc.log

```

these logs have three pipe lines(each for specific line type) , so we see 3 instances of logstash processing them .

here's the pipeline used

```auto
- pipeline.id: r_line
  pipeline.workers: 16
  pipeline.batch.size: 3000
  queue.type: memory
  path.config: "/etc/logstash/conf.d/tests/logstash-dissect_r_line.conf"
- pipeline.id: f_line
  pipeline.workers: 16
  pipeline.batch.size: 3000
  queue.type: memory
  path.config: "/etc/logstash/conf.d/tests/logstash-dissect_f_line.conf"
- pipeline.id: s_line
  pipeline.workers: 16
  pipeline.batch.size: 3000
  queue.type: memory
  path.config: "/etc/logstash/conf.d/tests/logstash-dissect_s_line.conf"

```

Here;s the input section that looks for specific line type - the log has 3 types in the same file -\> r/f/S lines

```auto
root@dev-elk-app02:/app/input# cat /etc/logstash/conf.d/tests/logstash-dissect_r_line_mle.conf 
input {
file {
path => "/app/input/*_ghostddc.log"
start_position => "beginning"
mode => "read"
close_older => 0
sincedb_path => "/dev/null"
check_archive_validity => "true"
stat_interval => "1ms"
discover_interval => 1
file_chunk_size => 3276800
max_open_files => 100000
file_completed_action => "log_and_delete"
file_completed_log_path => "/app/processed/output.txt"
}
#stdin {}
      }
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 19, 2022, 8:43am UTC](https://discuss.elastic.co/t/logstash-to-process-multiple-files-of-large-size/300299/2 "2022-04-19T08:43:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.