Problem:
I want filebeat only read the newly added line in log file and send it to logstash.
For example, there are 2lines in the log originally.
Line1: A
Line2: B
The log file will keep updating, says a new line3,
Line1: A
Line2: B
Line3: C
It expected only "Line3: C" will send to logstash, but filebeat will send Line1,2,3 again after new line 3 is added.
I then found suggestion about "tail_files":
https://www.elastic.co/guide/en/beats/filebeat/master/configuration-filebeat-options.html#_literal_type_literal
Stopped filebeat, deleted regiestry and start again, but it still not work.
version:
filebeat-5.6.3-linux-x86_64
filebeat.yml:
filebeat.prospectors:
-input_type: log
paths:
-/{path}/myLogFile.log
tail_files: truemultiline.pattern: (\n|^)[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9].[0-9][0-9][0-9]
multiline.negate: true
multiline.match: afteroutput.logstash:
hosts: ["Windows:5044"]
Also, i used command below for debugging
/{path}/filebeat -e -d "*" -c /filebeat.yml
After I update my log file for adding Line3, I found the offset value is reset to 0. I guess this is the reason why tail_files cant work.
2017/11/07 08:47:54.410566 prospector.go:183: DBG Run prospector
2017/11/07 08:47:54.410590 prospector_log.go:70: DBG Start next scan
2017/11/07 08:47:54.410650 prospector_log.go:226: DBG Check file for harvesting: /{path}/myLogFile.log
2017/11/07 08:47:54.410670 prospector_log.go:259: DBG Update existing file for harvesting: /{path}/myLogFile.log, offset: 340
2017/11/07 08:47:54.410679 prospector_log.go:311: DBG Harvester for file is still running: /{path}/myLogFile.log
2017/11/07 08:47:54.410690 prospector_log.go:91: DBG Prospector states cleaned up. Before: 2, After: 2
2017/11/07 08:47:59.403057 spooler.go:89: DBG Flushing spooler because of timeout. Events flushed: 0
2017/11/07 08:47:59.410997 log_file.go:84: DBG End of file reached: /{path}/myLogFile.log; Backoff now.
2017/11/07 08:48:04.403261 spooler.go:89: DBG Flushing spooler because of timeout. Events flushed: 0
2017/11/07 08:48:04.410848 prospector.go:183: DBG Run prospector
2017/11/07 08:48:04.410874 prospector_log.go:70: DBG Start next scan
2017/11/07 08:48:04.410933 prospector_log.go:226: DBG Check file for harvesting: /{path}/myLogFile.log
2017/11/07 08:48:04.410951 prospector_log.go:259: DBG Update existing file for harvesting: /{path}/myLogFile.log, offset: 340
2017/11/07 08:48:04.410961 prospector_log.go:311: DBG Harvester for file is still running: /{path}/myLogFile.log
2017/11/07 08:48:04.410972 prospector_log.go:91: DBG Prospector states cleaned up. Before: 2, After: 2
2017/11/07 08:48:09.403516 spooler.go:89: DBG Flushing spooler because of timeout. Events flushed: 0
2017/11/07 08:48:09.411273 log_file.go:84: DBG End of file reached: /{path}/myLogFile.log; Backoff now.
2017/11/07 08:48:14.365303 metrics.go:34: INFO No non-zero metrics in the last 30s
2017/11/07 08:48:14.403759 spooler.go:89: DBG Flushing spooler because of timeout. Events flushed: 0
2017/11/07 08:48:14.411117 prospector.go:183: DBG Run prospector
2017/11/07 08:48:14.411135 prospector_log.go:70: DBG Start next scan
2017/11/07 08:48:14.411184 prospector_log.go:226: DBG Check file for harvesting: /{path}/myLogFile.log
2017/11/07 08:48:14.411200 prospector_log.go:259: DBG Update existing file for harvesting: /{path}/myLogFile.log, offset: 0
2017/11/07 08:48:14.411208 prospector_log.go:268: DBG Resuming harvesting of file: /{path}/myLogFile.log, offset: 0
2017/11/07 08:48:14.411554 log.go:288: DBG Setting offset for file based on seek: /{path}/myLogFile.log
2017/11/07 08:48:14.411569 log.go:273: DBG Setting offset for file: /{path}/myLogFile.log. Offset: 0
2017/11/07 08:48:14.411616 prospector_log.go:91: DBG Prospector states cleaned up. Before: 2, After: 2
2017/11/07 08:48:14.411653 log.go:91: INFO Harvester started for file: /{path}/myLogFile.log
2017/11/07 08:48:14.411854 log_file.go:84: DBG End of file reached: /{path}/myLogFile.log; Backoff now.
2017/11/07 08:48:15.412080 log_file.go:84: DBG End of file reached: /{path}/myLogFile.log; Backoff now.
But why.gif?