I'm trying to connect to password secured remote FTP server where I have a few log files. I want to get a steady stream of new events to Logstash. As far as I know, Logstash can't connect directly to FTP server. Is there an easy way to do it?
I tried solution with 'exec' command, http://stackoverflow.com/questions/37394792/collect-log-files-from-ftp-into-logstash-elasticsearch , but this parses the file only once and doesn't remember the last read position in the file like 'file' plugin.
I decided to sync log files hourly from FTP server to my local computer and read those files in Logstash. The problem is, that when parsing files, Logstash doesn't stop when it reaches the end of file, but continues to send data from the beginning. Probably due to modification of files from syncing.
My 'file' plugin looks like this:
path => "C:/log1.log"
start_position => beginning
ignore_older => 0
Am I missing something here?
Is there any alternative solution for sending increment of logs from FTP server directly to Logstash?