Prevent re-reading files from beginning


(Mark Van Rossum) #1

The original "lumberjack" program which we are currently using, had an option "-from-beginning" which defaulted to false. This meant when you started "lumberjack" up, it wouldn't read the entire file, but only send new lines in the file.

It seems that in logstash-forwarder 0.4 this option has been removed, and (worse!) it now appears to default to "true". This means each time we redeploy logstash forwarder to the target VMs, it rereads and ships all the files contents every time so we get duplicate log entries.

Is there anyway to prevent this i.e. restore the original "-from-beginning=false" behaviour? Alternatively is there a way to specify the location of the ".logstash-forwarder" file that maintains it's state? Currently this file is removed every time we redeploy the agent. If we could specify the location we can put it somewhere on the filesystem where it won't get removed when we redeploy.


(Magnus Bäck) #2

LSF unconditionally opens .logstash-forwarder (below), but I can't find any chdir call so I'd assume that you can just change which directory you start LSF from and thereby indirectly affect the state file's path.


(Magnus Bäck) #3

(Mark Van Rossum) #4

That works OK, thanks!

Also I realised that the "-from-beginning" flag has been renamed "-tail", so setting "-tail=true" reinstates the original default behaviour


#5

where to specify this tail option? i'm starting logstash forwarder as a service.


(Mark Van Rossum) #6

You need to supply it to the command when you start it - you might need to edit your init scripts


#7

thank! Is there a doc on the available command line flags for LSF? I couldn't find any...


(Magnus Bäck) #8

Like most programs out there there's a command-line option that lists the available command-line options. In the LSF case it's -h.


#9

thank you


(system) #10