Problem with updating Filebeat 8.11.4 to 8.13.4 - unwanted retransmision of all files

Elastic Stack Beats
2

This really makes me believe that there is something on your prod environment that's making Filebeat use a different folder for the registry.

When Filebeat starts up it will log the folders it is using:

{
  "log.level": "info",
  "@timestamp": "2024-10-10T08:27:02.660-0400",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure",
    "file.name": "instance/beat.go",
    "file.line": 1058
  },
  "message": "Home path: [/tmp/go-build319583623/b001/exe] Config path: [/tmp/go-build319583623/b001/exe] Data path: [/tmp/go-build319583623/b001/exe/data] Logs path: [/tmp/go-build319583623/b001/exe/logs]",
  "service.name": "filebeat",
  "ecs.version": "1.6.0"
}

Check if the data path is correct and the files are there.

Another, more remote possibility is that something has changed with the file identity. Has anything changed in the host other than the Filebeat version?

By default Filebeat will use inode + deviced ID to identify files uniquely, depending on your OS/file system those can change in certain situations.

Here is an registry entry (${path.data}/registry/filebeat/log.json) from the log input:

{
  "k": "filebeat::logs::native::23201-39",
  "v": {
    "id": "native::23201-39",
    "prev_id": "",
    "offset": 0,
    "type": "log",
    "identifier_name": "native",
    "source": "/tmp/flog.log",
    "timestamp": [
      280445065136719,
      1728515609
    ],
    "ttl": -1,
    "FileStateOS": {
      "inode": 23201,
      "device": 39
    }
  }
}

If you look at "k": "filebeat::logs::native::23201-39", 23201 is the inode and 39 is the device ID. You can check if they have changed for your files.