Duplicate file entries in registry file on system with RAID 5

Hi,

Debian 10 linux with software raid 5, 3 ssd disks:

    ARRAY /dev/md1 level=raid1 num-devices=3 metadata=0.90 UUID=6a96a1c6:818e4a81:5ca9666b:e9209fa3
    ARRAY /dev/md2 level=raid1 num-devices=3 metadata=0.90 UUID=89ed1650:8b88611a:5ca9666b:e9209fa3
    ARRAY /dev/md3 level=raid1 num-devices=3 metadata=1.2 name=10933001:swap UUID=f80f1c0e:b0d75aef:6332d0e8:e13ec433
    ARRAY /dev/md4 level=raid5 num-devices=3 metadata=1.2 name=10933001:tmp UUID=34cdef57:64a1b74a:3ac42e3d:7edf3ced
    ARRAY /dev/md5 level=raid5 num-devices=3 metadata=1.2 name=10933001:root UUID=9d39c71a:a6794054:b091bfce:c0abbaad

filebeat config:

    filebeat.registry.flush: 60s
    filebeat.shutdown_timeout: 10s
    filebeat.inputs:
      - type: log
        clean_inactive: 720h
        ignore_older: 48h
        close_inactive: 1h
        scan_frequency: 30s
        paths:
          - /var/www/app/web/storage/parsed/*/*/*/*/logstash.log
    output.logstash:
      hosts: ["10.32.20.33:5043"]

after server reboot i find, that filebeat duplicated some files in registry with different device ids and also with different offsets, example:

    {
    	"_key": "filebeat::logs::native::22151340-2429",
    	"offset": 6058887,
    	"ttl": 2592000000000000,
    	"type": "log",
    	"FileStateOS": {
    		"inode": 22151340,
    		"device": 2429
    	},
    	"id": "native::22151340-2429",
    	"prev_id": "",
    	"source": "/var/www/app/web/storage/parsed/2021/05/12/574/logstash.log",
    	"timestamp": [516275907982, 1620833672],
    	"identifier_name": "native"
    }, {
    	"_key": "filebeat::logs::native::22151340-2430",
    	"source": "/var/www/app/web/storage/parsed/2021/05/12/574/logstash.log",
    	"timestamp": [516035301277, 1620833673],
    	"identifier_name": "native",
    	"id": "native::22151340-2430",
    	"prev_id": "",
    	"offset": 165418997,
    	"ttl": 2592000000000000,
    	"type": "log",
    	"FileStateOS": {
    		"inode": 22151340,
    		"device": 2430
    	}
    }

how it's possible that same file, with same inodeId, but different deviceIds and offset appear in registry?

probably someone met such situation.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.