The original "lumberjack" program which we are currently using, had an option "-from-beginning" which defaulted to false. This meant when you started "lumberjack" up, it wouldn't read the entire file, but only send new lines in the file.
It seems that in logstash-forwarder 0.4 this option has been removed, and (worse!) it now appears to default to "true". This means each time we redeploy logstash forwarder to the target VMs, it rereads and ships all the files contents every time so we get duplicate log entries.
Is there anyway to prevent this i.e. restore the original "-from-beginning=false" behaviour? Alternatively is there a way to specify the location of the ".logstash-forwarder" file that maintains it's state? Currently this file is removed every time we redeploy the agent. If we could specify the location we can put it somewhere on the filesystem where it won't get removed when we redeploy.