Filebeat sand data again in a file when restarting
paths property is setted with absolute path. It is work well( Restarting filbert, does not send duplicate data)
but setting with relative path, send duplicate data.
Is it bug?
filebeat.prospectors:
- input_type: log
paths:- /home1/irteam/datalog/*.log
...
output.console:
pretty: true
test input file
$ echo "test 1" >> /home1/irteam/datalog/test.log
$ echo "test 2" >> /home1/irteam/datalog/test.log
$ echo "test 3" >> /home1/irteam/datalog/test.log
result output : console output work well
$ ./filebeat -c ./filebeat.yml
{"@timestamp":"2017-10-25T05:32:23.693Z","beat":{"hostname":"Naverui-MacBook-Pro-182.local","name":"Naverui-MacBook-Pro-182.local","version":"5.6.0"},"input_type":"log","message":"test 1","offset":7,"source":"/Users/Naver/filebeat/beats/filebeat/datalog/test.log","type":"log"}
{"@timestamp":"2017-10-25T05:32:48.696Z","beat":{"hostname":"Naverui-MacBook-Pro-182.local","name":"Naverui-MacBook-Pro-182.local","version":"5.6.0"},"input_type":"log","message":"test 2","offset":14,"source":"/Users/Naver/filebeat/beats/filebeat/datalog/test.log","type":"log"}
{"@timestamp":"2017-10-25T05:32:48.696Z","beat":{"hostname":"Naverui-MacBook-Pro-182.local","name":"Naverui-MacBook-Pro-182.local","version":"5.6.0"},"input_type":"log","message":"test 3","offset":21,"source":"/Users/Naver/filebeat/beats/filebeat/datalog/test.log","type":"log"}
But setted with relative path like below.
appended event output work well but restarting filebeat, old data send again
filebeat.prospectors:
- input_type: log
paths:- ./datalog/*.log
$ echo "test 1" >> /home1/irteam/datalog/test.log
$ echo "test 2" >> /home1/irteam/datalog/test.log
$ echo "test 3" >> /home1/irteam/datalog/test.log
./filebeat -c ./filebeat.yml
{"@timestamp":"2017-10-25T05:42:09.732Z","beat":{"hostname":"Naverui-MacBook-Pro-182.local","name":"Naverui-MacBook-Pro-182.local","version":"5.6.0"},"input_type":"log","message":"test 1","offset":7,"source":"/Users/Naver/filebeat/beats/filebeat/datalog/test.log","type":"log"}
{"@timestamp":"2017-10-25T05:42:09.732Z","beat":{"hostname":"Naverui-MacBook-Pro-182.local","name":"Naverui-MacBook-Pro-182.local","version":"5.6.0"},"input_type":"log","message":"test 2","offset":14,"source":"/Users/Naver/filebeat/beats/filebeat/datalog/test.log","type":"log"}
{"@timestamp":"2017-10-25T05:42:09.732Z","beat":{"hostname":"Naverui-MacBook-Pro-182.local","name":"Naverui-MacBook-Pro-182.local","version":"5.6.0"},"input_type":"log","message":"test 3","offset":21,"source":"/Users/Naver/filebeat/beats/filebeat/datalog/test.log","type":"log"}