Validate file with logstash-file-input plugin

shabirmean · April 4, 2017, 1:32am

Hi,

I am working on a project that uses Logstash to parse some csv files. For this the Logstash “file-input-plugin” is used where new files are read from a path and the parsing is done.

However, I am trying to implement a validation to this flow: whenever a new file is captured, calculate its SHA DIGEST and call an endpoint and validate that it’s a valid file.

DISCLAIMER: My experience with Ruby is NIL . Mostly accustomed to Java.

Regardless, I was going through the logstash file-input-plugin source (https://github.com/logstash-plugins/logstash-input-file) and I saw that they had used a package named “filewatch/tail”. Searching for it lead me to this repo: https://github.com/jordansissel/ruby-filewatch

I found that they call a method “subscribe” of FileWatch::Tail at

github.com

logstash-plugins/logstash-input-file/blob/master/lib/logstash/inputs/file.rb#L305


  stop
  # use observer listener api
  @tail = FileWatch::Tail.new_observing(@tail_config)
  @tail.logger = @logger
  @path.each { |path| @tail.tail(path) }
end


def run(queue)
  begin_tailing
  @queue = queue
  @tail.subscribe(self)
  exit_flush
end # def run


def post_process_this(event)
  event.set("[@metadata][host]", @host)
  event.set("host", @host) if !event.include?("host")
  decorate(event)
  @queue << event
end

Can I get some guidance as to where exactly can I write into something to have the custom logic – calculate file DIGEST -> Call endpoint and validate -> if valid then process file else raise an error – whenever a new file is grabbed?

Your time and guidance will be much appreciated.

Thanks
Shabir

guyboertje · April 5, 2017, 12:50pm

@shabirmean
I wrote those changes to the file input.
The file input does not have a facility for this - the part you would want to hook into (but can't now) is the file discovery loop which scans for files to the 'to-be-processed' collection.

The only way you can do this is to script something yourself using a staging folder...

new files go into a staging folder -> staging
a script monitors for new files, does the digest + compare.
move file to a valid folder -> valid
LS detects the file in the valid folder and begins tailing it.

Topic		Replies	Views
Where to have logic to run everytime a new event occurs Logstash	3	520	May 8, 2017
Logstash does not parse all logs with file as input Logstash	2	250	July 13, 2021
File input plugin 4.1.2 [filewatch.tailmode.handlers.shrink] mismatch on sincedb_value.watched_file, this should have been handled by Discoverer Logstash	1	268	July 4, 2018
Process downloaded log files Logstash	11	1703	June 20, 2019
Do something when Logstash has finished processing Logstash	3	1287	June 19, 2018

Validate file with logstash-file-input plugin

Related topics