Connect Logstash to remote FTP server

[Nejc_Gutnik]

Hi there,

I'm trying to connect to password secured remote FTP server where I have a few log files. I want to get a steady stream of new events to Logstash. As far as I know, Logstash can't connect directly to FTP server. Is there an easy way to do it?

I tried solution with 'exec' command, http://stackoverflow.com/questions/37394792/collect-log-files-from-ftp-into-logstash-elasticsearch , but this parses the file only once and doesn't remember the last read position in the file like 'file' plugin.

I decided to sync log files hourly from FTP server to my local computer and read those files in Logstash. The problem is, that when parsing files, Logstash doesn't stop when it reaches the end of file, but continues to send data from the beginning. Probably due to modification of files from syncing.

My 'file' plugin looks like this:

file {
path => "C:/log1.log"
start_position => beginning
ignore_older => 0
}

Am I missing something here?
Is there any alternative solution for sending increment of logs from FTP server directly to Logstash?

Thanks,
Nejc

[magnusbaeck (Magnus Bäck)]

As far as I know, Logstash can't connect directly to FTP server. Is there an easy way to do it?

I don't think there is an FTP input plugin, so apart from what you're already doing there isn't much you can do apart from writing your own input plugin.

I decided to sync log files hourly from FTP server to my local computer and read those files in Logstash. The problem is, that when parsing files, Logstash doesn't stop when it reaches the end of file, but continues to send data from the beginning. Probably due to modification of files from syncing.

Yeah, if you overwrite the files Logstash loses track.

Is there any alternative solution for sending increment of logs from FTP server directly to Logstash?

I suspect you'll have to write a script that basically diffs new logfiles against old ones and copies the new lines to a new file that Logstash can pick up.