Rsync changing inode numbers causing multiple imports

Elastic Stack Logstash
1

Hi All,

Fairly new to the whole ELK stack and running into an issue.

To test things out I am rsync'ing a bunch of logs from various Windows IIS servers back to my host running the ELK software.

Things are working fine, but because rsync creates a new file and a new inode each time I am getting the same logs imported each time.

Googling found me this 'patch' that seems to address this exact issue:
https://github.com/michio-nikaido/ruby-filewatch/commit/a376432a64a821a569490d0b15eb4d0060b8f60f

How do I go about getting this into by currently running logstash?

Running logstash 1.5.3 on a Fedora box (to do a proof of concept).

Not a ruby person, so not sure how I can update just this package to get this additional functionality.

Thanks,
Jason

2

This is, unfortunately, a short coming of the file watch library.

I've raised a GH issue here for it :slight_smile:

3

Rsync has an --inplace option that will overwrite the existing file instead of making a new copy and moving it. There is a warning in the manpage though:

WARNING: The file's data will be in an inconsistent state during the transfer (and possibly afterward if the transfer gets interrupted), so you should not use this option to update files that are in use. Also note that rsync will be unable to update a file in-place that is not writable by the receiving user.

I think that so long as the file only changes in an append-only way, and a simple test shows this method as working