Logstash to process multiple files of large size?

Elastic Stack Logstash
1

Hello,

To give a little bit of context, i have 3 VMs , each of 16 CPU cores out of which 2 are of 16GB RAM (they host Elasticsearch and kibana + a flask app, that provides a front end for the user to input the machineip to fetch logs )

The VM with logstash has 24GB of RAM . these Logs are of variable sizes , sometimes as large as 15GB and some as small as 150MB/1GB . my question is logstash seems to process one log at a time even though there are multiple files

Here;s lsof output that shows logstash processing 1 file, even though there are 2 .

root@dev-elk-app02:/app/input# lsof *
COMMAND  PID USER   FD   TYPE DEVICE    SIZE/OFF   NODE NAME
java    7218 root  118r   REG  253,0 38579356903 131074 22.223.11.5_B-V-43YX5ZH_ddc.log
java    7218 root  119r   REG  253,0 38579356903 131074 22.223.11.5_B-V-43YX5ZH_ddc.log
java    7218 root  120r   REG  253,0 38579356903 131074 22.223.11.5_B-V-43YX5ZH_ddc.log
root@dev-elk-app02:/app/input# ls
189.21.70.65_ddc.log	 22.223.11.5_ddc.log 20.221.11.46_ghostddc.log

these logs have three pipe lines(each for specific line type) , so we see 3 instances of logstash processing them .

here's the pipeline used

- pipeline.id: r_line
  pipeline.workers: 16
  pipeline.batch.size: 3000
  queue.type: memory
  path.config: "/etc/logstash/conf.d/tests/logstash-dissect_r_line.conf"
- pipeline.id: f_line
  pipeline.workers: 16
  pipeline.batch.size: 3000
  queue.type: memory
  path.config: "/etc/logstash/conf.d/tests/logstash-dissect_f_line.conf"
- pipeline.id: s_line
  pipeline.workers: 16
  pipeline.batch.size: 3000
  queue.type: memory
  path.config: "/etc/logstash/conf.d/tests/logstash-dissect_s_line.conf"

Here;s the input section that looks for specific line type - the log has 3 types in the same file -> r/f/S lines

root@dev-elk-app02:/app/input# cat /etc/logstash/conf.d/tests/logstash-dissect_r_line_mle.conf 
input {
file  {
path => "/app/input/*_ghostddc.log"
start_position => "beginning"
mode => "read"
close_older => 0
sincedb_path => "/dev/null"
check_archive_validity => "true"
stat_interval => "1ms"
discover_interval => 1
file_chunk_size => 3276800
max_open_files => 100000
file_completed_action => "log_and_delete"
file_completed_log_path => "/app/processed/output.txt"
}
#stdin {}
      }
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.