Remote file collection (rsync, scp, ...)

Elastic Stack Logstash
1

I need to collect log files from a closed appliance. I cannot configure the appliance to send log events out with syslog. I cannot install software on the appliance. I can copy files from the appliance, e.g. with rsync or scp.

The log files rotate from a basename (logfile.log) to an archived name (logfile_timestamp.log).

I was planning to frequently rsync the files to a linux box and use filebeat to ship them to elastic.

However, some reading suggests that logstash/filebeat + rsync does not work reliably (e.g. due to rsync changing inode and logstash/filebeat depending on that staying the same).

For scp, "When copying a source file to a target file which already exists, scp will replace the contents of the target file (keeping the inode)." So I think that might be the non-optimal solution.

Is there another solution that I'm missing?

1 Like
2

For scp, "When copying a source file to a target file which already exists, scp will replace the contents of the target file (keeping the inode)." So I think that might be the non-optimal solution.

Does scp truncate the file first or does it overwrite the file in place? In the latter case you'll probably be fine, otherwise you'll probably need to write a script that runs after the rsync operation and either appends new lines to the files that Filebeat monitors or overwrites them in place.