How to recycle the file after reading with logstash-input-file plugin

Elastic Stack Logstash
1

Hi,
I have a logstash configuration as below. I'm expecting the json file is moved to archive path when reading is completed, but the json file is always there, and archive path is empty. Do I miss out anything in configuration?

input{
	
	file {
		mode => "read"
		path => ["c:/EDM/tools/dremio_dashboard/sys_jobs_recent.json"]
		file_completed_action => "log_and_delete"
		file_completed_log_path => "c:/EDM/tools/dremio_dashboard/archive"
		sincedb_clean_after => "1 day"
		start_position => "beginning"
		stat_interval => "1 hour"
		codec => json
	}
}

filter {
}

output {
 
  stdout {
    codec => rubydebug {}
  }
  elasticsearch {
	hosts => ["http://localhost:9200"]
	index => "dremio_sys_jobs_recent"
	ilm_enabled => "true"
	ilm_rollover_alias => "dremio_sys_jobs_data"
	ilm_policy => "dremio_policy"
  }	
}
2

Hi @chun,

You are using the log_and_delete configuration. This option is designed to delete the file after processing, but it will not move the file to another location.

Link: File input plugin | Logstash Reference [8.15] | Elastic

3

The file input expects to write log entries to the file specified by file_completed_log_path. If it points to a directory instead of a file then the path file is read, but it is not deleted and the action is not logged.

4

Yes, the file is deleted after I change file_completed_log_path point to a log file. Thanks so much!

1 Like