Validate file with logstash-file-input plugin

shabirmean · April 4, 2017, 1:32am

Hi,

I am working on a project that uses Logstash to parse some csv files. For this the Logstash “file-input-plugin” is used where new files are read from a path and the parsing is done.

However, I am trying to implement a validation to this flow: whenever a new file is captured, calculate its SHA DIGEST and call an endpoint and validate that it’s a valid file.

DISCLAIMER: My experience with Ruby is NIL . Mostly accustomed to Java.

Regardless, I was going through the logstash file-input-plugin source (https://github.com/logstash-plugins/logstash-input-file) and I saw that they had used a package named “filewatch/tail”. Searching for it lead me to this repo: https://github.com/jordansissel/ruby-filewatch

I found that they call a method “subscribe” of FileWatch::Tail at

github.com

logstash-plugins/logstash-input-file/blob/master/lib/logstash/inputs/file.rb#L305


  stop
  # use observer listener api
  @tail = FileWatch::Tail.new_observing(@tail_config)
  @tail.logger = @logger
  @path.each { |path| @tail.tail(path) }
end


def run(queue)
  begin_tailing
  @queue = queue
  @tail.subscribe(self)
  exit_flush
end # def run


def post_process_this(event)
  event.set("[@metadata][host]", @host)
  event.set("host", @host) if !event.include?("host")
  decorate(event)
  @queue << event
end

Can I get some guidance as to where exactly can I write into something to have the custom logic – calculate file DIGEST -> Call endpoint and validate -> if valid then process file else raise an error – whenever a new file is grabbed?

Your time and guidance will be much appreciated.

Thanks
Shabir

guyboertje · April 5, 2017, 12:50pm

@shabirmean
I wrote those changes to the file input.
The file input does not have a facility for this - the part you would want to hook into (but can't now) is the file discovery loop which scans for files to the 'to-be-processed' collection.

The only way you can do this is to script something yourself using a staging folder...

new files go into a staging folder -> staging
a script monitors for new files, does the digest + compare.
move file to a valid folder -> valid
LS detects the file in the valid folder and begins tailing it.

system · May 3, 2017, 12:50pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.