How to get logstash to read/process multiple log files at the same time?

Elastic Stack Logstash
1

It seems that logstash doesnt read multiple files in the directory at the same time , what am i missing ? how do i get logstash to read all three files here ?

root@dev-elk-app02:/app/input# lsof *
COMMAND  PID USER   FD   TYPE DEVICE    SIZE/OFF   NODE NAME
java    7218 root  118r   REG  253,0 38579356903 131074 22.223.11.125_ddc.log
java    7218 root  119r   REG  253,0 38579356903 131074 22.223.11.125_ddc.log
java    7218 root  120r   REG  253,0 38579356903 131074 22.223.11.125_ddc.log
root@dev-elk-app02:/app/input# ls
189.21.70.65_ddc.log	 22.223.11.125_ddc.log 20.221.11.46_ddc.log

Here's a sample input

root@dev-elk-app02:/app/input# cat /etc/logstash/conf.d/tests/logstash-dissect_r_line_mle.conf 
input {
file  {
path => "/app/input/*_ddc.log"
start_position => "beginning"
mode => "read"
close_older => 0
sincedb_path => "/dev/null"
check_archive_validity => "true"
stat_interval => "1ms"
discover_interval => 1
file_chunk_size => 3276800
max_open_files => 100000
file_completed_action => "log_and_delete"
file_completed_log_path => "/app/processed/output.txt"
}
#stdin {}
      }
2

If you enable log.level trace you should see which files it is checking/opening/closing.

3

How are you starting Logstash?

This is most likely due to the user permissions on each file. Currently it is root but if you are starting Logstash as a service it should be logstash. Or the group.

If you are starting Logstash via command line you might just need to add sudo.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.