Hi,
our Filebeat (7.6.1 currently, it was an older version but we've upgraded it to the newest one for testing purposes) is running as a sidecar container. We have the data folder persisted. The data/registry/filebeat/data.json is being constantly updated/changed. Filebeat is setup to harvest logs from only one file (which is also persisted). Everything works fine until the pod is restarted for any reason. After the new pod starts running it pushes all the older logs from that file to Logstash (we see them in Kibana) and then it just goes normal again. (As if we have not persisted the registry directory).
We've added a test file and added it as an input. The file's content is: {"message": "line 1","time": "12:01"} {"message": "line 2","time": "12:02"} {"message": "line 3","time": "12:03"} {"message": "line 4","time": "12:04"}
The same goes for these lines, once the pod restarts, all of these log lines get duplicated on Kibana.
I must emphasize that the content of data/registry/filebeat/data.json was changed after we've added the second (test) input from : {"source":"/usr/share/filebeat/foo/bar/json/foo.json","offset":9909624,"timestamp":"2020-03-18T15:22:29.952702443Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":foo,"device":bar}} to [{"source":"/usr/share/filebeat/foo/bar/json/testlogs.json","offset":152,"timestamp":"2020-03-18T15:21:19.535225865Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":foo,"device":bar}},{"source":"/usr/share/filebeat/foo/bar/json/foo.json","offset":9909624,"timestamp":"2020-03-18T15:22:29.952702443Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":foo,"device":bar}}] (Timestamps were different, this is just a template) what we see as a valid configuration of filebeat.registry.path. One thing more, the value of "offset": for test logs is 152 what is 38*4 where 38 is the number of characters in each log line (the last character of the file) which would indicate that registry file knows that there are no new logs in that file.
Do you have any suggestions?