Prevent re-reading files from beginning

Mark_van_Rossum · July 20, 2015, 10:25am

The original "lumberjack" program which we are currently using, had an option "-from-beginning" which defaulted to false. This meant when you started "lumberjack" up, it wouldn't read the entire file, but only send new lines in the file.

It seems that in logstash-forwarder 0.4 this option has been removed, and (worse!) it now appears to default to "true". This means each time we redeploy logstash forwarder to the target VMs, it rereads and ships all the files contents every time so we get duplicate log entries.

Is there anyway to prevent this i.e. restore the original "-from-beginning=false" behaviour? Alternatively is there a way to specify the location of the ".logstash-forwarder" file that maintains it's state? Currently this file is removed every time we redeploy the agent. If we could specify the location we can put it somewhere on the filesystem where it won't get removed when we redeploy.

magnusbaeck · July 20, 2015, 2:39pm

LSF unconditionally opens .logstash-forwarder (below), but I can't find any chdir call so I'd assume that you can just change which directory you start LSF from and thereby indirectly affect the state file's path.

github.com

elastic/logstash-forwarder/blob/v0.4.0/logstash-forwarder.go#L165


	// - publisher: writes to the network, notifies registrar
	// - registrar: records positions of files read
	// Finally, prospector uses the registrar information, on restart, to
	// determine where in each file to restart a harvester.


	restart := &ProspectorResume{}
	restart.persist = make(chan *FileState)


	// Load the previous log file locations now, for use in prospector
	restart.files = make(map[string]*FileState)
	if existing, e := os.Open(".logstash-forwarder"); e == nil {
		defer existing.Close()
		wd := ""
		if wd, e = os.Getwd(); e != nil {
			emit("WARNING: os.Getwd retuned unexpected error %s -- ignoring\n", e.Error())
		}
		emit("Loading registrar data from %s/.logstash-forwarder\n", wd)


		decoder := json.NewDecoder(existing)
		decoder.Decode(&restart.files)
	}

magnusbaeck · July 20, 2015, 3:08pm

Mark_van_Rossum · July 22, 2015, 3:33pm

That works OK, thanks!

Also I realised that the "-from-beginning" flag has been renamed "-tail", so setting "-tail=true" reinstates the original default behaviour

zpp · July 28, 2015, 1:58pm

where to specify this tail option? i'm starting logstash forwarder as a service.

Mark_van_Rossum · July 29, 2015, 8:24pm

You need to supply it to the command when you start it - you might need to edit your init scripts

zpp · July 31, 2015, 7:48am

thank! Is there a doc on the available command line flags for LSF? I couldn't find any...

magnusbaeck · July 31, 2015, 6:14pm

Like most programs out there there's a command-line option that lists the available command-line options. In the LSF case it's -h.

zpp · August 3, 2015, 2:23am

thank you

Topic		Replies	Views
Prevent logstash from reading file from beginning when restarted Logstash	1	267	March 23, 2020
File input processing question Logstash	3	1077	July 6, 2017
Reading file from beginning using file input Logstash	6	3307	March 13, 2019
Multiple logstash forwarder instances Logstash	12	2233	July 6, 2017
How To Make File Input Read Entire File Every Time? Logstash	5	4826	July 6, 2017

Prevent re-reading files from beginning

Related topics