Data duplication problem after server migration

Elastic Stack Beats
1

Hi There:
Currently, we are using Filebeat (version 7.15.2) to harvest logs of program A and send them to Kafka. Due to some reason, we will do some operations called Server Migration freqently.

Server Migration means:

  1. Program A and Filebeat run on server A.
  2. Stop Filebeat and program A on server A, copy all the logs and configurations of Program A, > Filebeat, Filebeat data, Filebeat config, and Filebeat logs to server B, ensuring that the file's last modification time remains unchanged.
  3. Restart Filebeat and Program A on server B.

After the server migration, we observed that Filebeat harvest logs which have been harvested on server A. After reading filebeat logs, we found that this issue may be related to the different input_id which Filebeat generated on server A and server B separately.
Is there any solution to avoid this data duplication problem after server migration?

2

Here is the filebeat log.json

{"op":"set","id":30}
{"k":"filebeat::logs::native::196608-45870-3781051510","v":{"prev_id":"","source":"D:\\filebeatall\\test\\untiltest.txt","offset":2026973,"timestamp":[2061775969080,1690194390],"ttl":-1,"type":"log","id":"native::196608-45870-3781051510","FileStateOS":{"idxhi":196608,"idxlo":45870,"vol":3781051510},"identifier_name":"native"}}
{"op":"set","id":31}
{"k":"filebeat::logs::native::196608-45870-3781051510","v":{"offset":2026973,"timestamp":[2061775969080,1690194390],"type":"log","prev_id":"","source":"D:\\filebeatall\\test\\untiltest.txt","ttl":-1,"FileStateOS":{"idxhi":196608,"idxlo":45870,"vol":3781051510},"identifier_name":"native","id":"native::196608-45870-3781051510"}}
{"op":"set","id":32}
{"k":"filebeat::logs::native::196608-45870-3781051510","v":{"source":"D:\\filebeatall\\test\\untiltest.txt","offset":2026973,"ttl":-1,"identifier_name":"native","prev_id":"","timestamp":[2061668698380,1690194516],"type":"log","FileStateOS":{"idxhi":196608,"idxlo":45870,"vol":3781051510},"id":"native::196608-45870-3781051510"}}
{"op":"set","id":33}
{"k":"filebeat::logs::native::196608-45870-3781051510","v":{"prev_id":"","source":"D:\\filebeatall\\test\\untiltest.txt","offset":2026973,"ttl":-1,"type":"log","id":"native::196608-45870-3781051510","timestamp":[2061668698380,1690194516],"FileStateOS":{"idxhi":196608,"idxlo":45870,"vol":3781051510},"identifier_name":"native"}}
{"op":"set","id":34}
{"k":"filebeat::logs::native::65536-1755945-118598","v":{"timestamp":[2061669697280,1690194516],"ttl":-1,"type":"log","identifier_name":"native","id":"native::65536-1755945-118598","prev_id":"","offset":0,"source":"D:\\filebeatall\\test\\untiltest.txt","FileStateOS":{"idxhi":65536,"idxlo":1755945,"vol":118598}}}
{"op":"remove","id":35}
{"k":"filebeat::logs::native::196608-45870-3781051510"}
{"op":"set","id":36}
{"k":"filebeat::logs::native::65536-1755945-118598","v":{"prev_id":"","offset":0,"FileStateOS":{"vol":118598,"idxhi":65536,"idxlo":1755945},"identifier_name":"native","id":"native::65536-1755945-118598","source":"D:\\filebeatall\\test\\untiltest.txt","timestamp":[2061669697280,1690194516],"ttl":-1,"type":"log"}}
{"op":"set","id":37}
{"k":"filebeat::logs::native::65536-1755945-118598","v":{"prev_id":"","source":"D:\\filebeatall\\test\\untiltest.txt","type":"log","FileStateOS":{"idxhi":65536,"idxlo":1755945,"vol":118598},"id":"native::65536-1755945-118598","offset":81453,"timestamp":[2061823930080,1690194564],"ttl":-1,"identifier_name":"native"}}
{"op":"set","id":38}
{"k":"filebeat::logs::native::65536-1755945-118598","v":{"prev_id":"","source":"D:\\filebeatall\\test\\untiltest.txt","offset":180721,"ttl":-1,"type":"log","id":"native::65536-1755945-118598","timestamp":[2062120246480,1690194564],"FileStateOS":{"idxhi":65536,"idxlo":1755945,"vol":118598},"identifier_name":"native"}}
{"op":"set","id":39}
{"k":"filebeat::logs::native::65536-1755945-118598","v":{"source":"D:\\filebeatall\\test\\untiltest.txt","offset":301156,"timestamp":[2062312176280,1690194564],"FileStateOS":{"idxhi":65536,"idxlo":1755945,"vol":118598},"identifier_name":"native","id":"native::65536-1755945-118598","prev_id":"","ttl":-1,"type":"log"}}
{"op":"set","id":40}
{"k":"filebeat::logs::native::65536-1755945-118598","v":{"identifier_name":"native","id":"native::65536-1755945-118598","prev_id":"","offset":393316,"type":"log","FileStateOS":{"idxhi":65536,"idxlo":1755945,"vol":118598},"source":"D:\\filebeatall\\test\\untiltest.txt","timestamp":[2062530576380,1690194564],"ttl":-1}}
{"op":"set","id":41}
{"k":"filebeat::logs::native::65536-1755945-118598","v":{"type":"log","identifier_name":"native","source":"D:\\filebeatall\\test\\untiltest.txt","offset":485476,"timestamp":[2061814945280,1690194565],"ttl":-1,"FileStateOS":{"idxhi":65536,"idxlo":1755945,"vol":118598},"id":"native::65536-1755945-118598","prev_id":""}}

The k changed from filebeat::logs::native::196608-45870-3781051510 to filebeat::logs::native::65536-1755945-118598, and offset changed from 2026973 to 0 after server migration.

3

Yes, this is in the documentation.

Each filestream input must have a unique ID. Omitting or changing the filestream ID may cause data duplication. Without a unique ID, filestream is unable to correctly track the state of files.

You need to add an id to your input and use the same id in both servers.

4

Thansk for your reply.
Unfortunately, I have read this doc and tried before, so even we update the filebeat to 8.8 and use the filestream input plugin with the same id in both servers. This issue still occur.

Here is the filebeat log.json:

{"op":"set","id":14}
{"k":"filestream::my-streamfile-id::native::196608-45870-3781051510","v":{"ttl":1800000000000,"updated":[2062139226580,1690267384],"cursor":{"offset":973528},"meta":{"source":"D:\\filebeatall\\test\\untiltest.txt","identifier_name":"native"}}}
{"op":"set","id":15}
{"k":"filestream::my-streamfile-id::native::196608-45870-3781051510","v":{"cursor":{"offset":1054602},"meta":{"source":"D:\\filebeatall\\test\\untiltest.txt","identifier_name":"native"},"ttl":1800000000000,"updated":[2062325581080,1690267384]}}
{"op":"set","id":16}
{"k":"filestream::my-streamfile-id::native::196608-45870-3781051510","v":{"cursor":{"offset":1135894},"meta":{"source":"D:\\filebeatall\\test\\untiltest.txt","identifier_name":"native"},"ttl":1800000000000,"updated":[2062491778080,1690267384]}}
{"op":"set","id":17}
{"k":"filestream::my-streamfile-id::native::65536-6772-2758789039","v":{"ttl":0,"updated":[281470681743360,18446744011573954816],"cursor":null,"meta":{"source":"D:\\filebeatall\\test\\untiltest.txt","identifier_name":"native"}}}
{"op":"set","id":18}
{"k":"filestream::my-streamfile-id::native::65536-6772-2758789039","v":{"ttl":1800000000000,"updated":[2061588208280,1690267427],"cursor":null,"meta":{"identifier_name":"native","source":"D:\\filebeatall\\test\\untiltest.txt"}}}
{"op":"set","id":19}
{"k":"filestream::my-streamfile-id::native::65536-6772-2758789039","v":{"ttl":1800000000000,"updated":[2061927075480,1690267427],"cursor":{"offset":81453},"meta":{"source":"D:\\filebeatall\\test\\untiltest.txt","identifier_name":"native"}}}
{"op":"set","id":20}
{"k":"filestream::my-streamfile-id::native::65536-6772-2758789039","v":{"ttl":1800000000000,"updated":[2062287427080,1690267427],"cursor":{"offset":180721},"meta":{"source":"D:\\filebeatall\\test\\untiltest.txt","identifier_name":"native"}}}
{"op":"set","id":21}
{"k":"filestream::my-streamfile-id::native::65536-6772-2758789039","v":{"updated":[2061761059880,1690267428],"cursor":{"offset":301156},"meta":{"source":"D:\\filebeatall\\test\\untiltest.txt","identifier_name":"native"},"ttl":1800000000000}}

As you can see, while my-streamfile-id is the same id in both servers, The k changed from filestream::my-streamfile-id::native::196608-45870-3781051510 in server A to filestream::my-streamfile-id::native::65536-6772-2758789039 in server B, the offset changed from 1135894 in server A to 0 in server B.

5

For more information, here is the log that filebeat started in server B:

{"log.level":"info","@timestamp":"2023-07-25T14:43:43.598+0800","log.origin":{"file.name":"instance/beat.go","file.line":779},"message":"Home path: [D:\\filebeatall\\filebeat-id] Config path: [D:\\filebeatall\\filebeat-id] Data path: [D:\\filebeatall\\filebeat-id\\data] Logs path: [D:\\filebeatall\\filebeat-id\\logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:43.599+0800","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":870},"message":"Beat metadata path: D:\\filebeatall\\filebeat-id\\data\\meta.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:43.599+0800","log.origin":{"file.name":"instance/beat.go","file.line":787},"message":"Beat ID: 210d49b3-7f9c-4f39-a476-ed01f06c67da","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:43.605+0800","log.origin":{"file.name":"instance/beat.go","file.line":816},"message":"Set max procs limit: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:43.613+0800","log.logger":"processors","log.origin":{"file.name":"processors/processor.go","file.line":114},"message":"Generated new processors: add_fields={\"host\":{\"public_ip\":\"220.250.27.220\"}}, add_locale=[format=offset], add_host_metadata=[netinfo.enabled=[true], cache.ttl=[5m0s]]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:43.613+0800","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":96},"message":"Syscall filtering is only supported on Linux","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:43.618+0800","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1299},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"D:\\filebeatall\\filebeat-id","data":"D:\\filebeatall\\filebeat-id\\data","home":"D:\\filebeatall\\filebeat-id","logs":"D:\\filebeatall\\filebeat-id\\logs"},"type":"filebeat","uuid":"210d49b3-7f9c-4f39-a476-ed01f06c67da"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-07-25T14:43:43.619+0800","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1308},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"92c6b2370e46e549acda91b396f665a7e51e249c","libbeat":"8.8.2","time":"2023-06-23T19:04:45.000Z","version":"8.8.2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-07-25T14:43:43.619+0800","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1311},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"windows","arch":"amd64","max_procs":1,"version":"go1.19.10"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-07-25T14:43:43.623+0800","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1317},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-07-25T10:21:27+08:00","name":"KVM-XXXXXX","ip":["fe80::789d:6379:2985:47e2","172.24.131.126","::1","127.0.0.1","fe80::5efe:ac18:837e"],"kernel_version":"6.1.7601.24545 (win7sp1_ldr_escrow.200102-1707)","mac":["02:00:00:01:ef:14","00:00:00:00:00:00:00:e0"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows Server 2008 R2 Enterprise","version":"6.1","major":6,"minor":1,"patch":0,"build":"7601.24544"},"timezone":"CST","timezone_offset_sec":28800,"id":"cad5361a-c37e-48c6-b4b6-a9ce7713c726"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-07-25T14:43:43.624+0800","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1346},"message":"Process info","service.name":"filebeat","system_info":{"process":{"cwd":"D:\\filebeatall\\filebeat-id","exe":"D:\\filebeatall\\filebeat-id\\filebeat.exe","name":"filebeat.exe","pid":9108,"ppid":9052,"start_time":"2023-07-25T14:43:43.338+0800"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-07-25T14:43:43.624+0800","log.origin":{"file.name":"instance/beat.go","file.line":330},"message":"Setup Beat: filebeat; Version: 8.8.2","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:43.627+0800","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":358},"message":"Initializing output plugins","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:44.966+0800","log.logger":"kafka","log.origin":{"file.name":"kafka/kafka.go","file.line":48},"message":"initialize kafka output","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:44.966+0800","log.logger":"publisher","log.origin":{"file.name":"pipeline/consumer.go","file.line":110},"message":"start pipeline event consumer","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:44.966+0800","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":105},"message":"Beat name: KVM-XXXXXX","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:44.966+0800","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-07-25T14:43:44.966+0800","log.origin":{"file.name":"beater/filebeat.go","file.line":175},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:44.966+0800","log.origin":{"file.name":"instance/beat.go","file.line":516},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:44.966+0800","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":287},"message":"isFile(D:\\filebeatall\\filebeat-id\\data\\registry\\) -> false","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:44.966+0800","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":287},"message":"isFile() -> false","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:44.966+0800","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":280},"message":"isDir(D:\\filebeatall\\filebeat-id\\data\\registry\\filebeat) -> true","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:44.966+0800","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":287},"message":"isFile(D:\\filebeatall\\filebeat-id\\data\\registry\\filebeat\\meta.json) -> true","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:44.966+0800","log.logger":"registrar","log.origin":{"file.name":"registrar/migrate.go","file.line":82},"message":"Registry type '1' found","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:44.966+0800","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for 'D:\\filebeatall\\filebeat-id\\data\\registry\\filebeat'. Active transaction id=16","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:44.990+0800","log.logger":"service","log.origin":{"file.name":"service/service_windows.go","file.line":110},"message":"Windows is interactive: true","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:44.990+0800","log.logger":"publisher","log.origin":{"file.name":"pipeline/queue_reader.go","file.line":49},"message":"pipeline event consumer queue reader: start","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:44.991+0800","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":145},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:44.999+0800","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for 'D:\\filebeatall\\filebeat-id\\data\\registry\\filebeat'. Active transaction id=16","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-07-25T14:43:44.999+0800","log.origin":{"file.name":"beater/filebeat.go","file.line":307},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform request:append","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform request:delete","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform request:set","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform response:append","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform response:delete","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform response:set","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform pagination:append","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform pagination:delete","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform pagination:set","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":84},"message":"registering encoder 'application/json': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":87},"message":"registering encoder 'application/x-www-form-urlencoded': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":93},"message":"registering decoder 'application/json': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":96},"message":"registering decoder 'application/x-ndjson': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":99},"message":"registering decoder 'text/csv': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":102},"message":"registering decoder 'application/zip': returned error: <nil>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:45.001+0800","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.001+0800","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":140},"message":"Starting Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:45.001+0800","log.origin":{"file.name":"cfgfile/reload.go","file.line":163},"message":"Config reloader started","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"cfgfile","log.origin":{"file.name":"cfgfile/reload.go","file.line":193},"message":"Scan for new config files","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"cfgfile","log.origin":{"file.name":"cfgfile/cfgfile.go","file.line":204},"message":"Load config from file: D:\\filebeatall\\filebeat-id/input.yml","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"cfgfile","log.origin":{"file.name":"cfgfile/reload.go","file.line":212},"message":"Number of module configs found: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":90},"message":"Starting reload procedure, current runners: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":108},"message":"Start list: 1, Stop list: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"cfgwarn","log.origin":{"file.name":"cfgtype/byte_size.go","file.line":36},"message":"DEPRECATED: size now requires a unit (KiB, MiB, etc...), current value: 204800. Will be removed in version: 7.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":286},"message":"recursive glob enabled","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":141},"message":"Starting runner: filestream","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":120},"message":"Input 'filestream' starting","service.name":"filebeat","id":"my-streamfile-id","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"input.filestream","log.origin":{"file.name":"filestream/prospector.go","file.line":131},"message":"Starting prospector","service.name":"filebeat","id":"my-streamfile-id","prospector":"file_prospector","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":139},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":213},"message":"Found 1 paths","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"input.filestream","log.origin":{"file.name":"filestream/prospector.go","file.line":177},"message":"A new file D:\\filebeatall\\test\\untiltest.txt has been found","service.name":"filebeat","id":"my-streamfile-id","prospector":"file_prospector","operation":"create","source_name":"native::65536-6772-2758789039","os_id":"65536-6772-2758789039","new_path":"D:\\filebeatall\\test\\untiltest.txt","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"input.filestream","log.origin":{"file.name":"input-logfile/harvester.go","file.line":138},"message":"Starting harvester for file","service.name":"filebeat","id":"my-streamfile-id","source_file":"filestream::my-streamfile-id::native::65536-6772-2758789039","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":176},"message":"newLogFileReader with config.MaxBytes:10485760","service.name":"filebeat","id":"my-streamfile-id","source_file":"filestream::my-streamfile-id::native::65536-6772-2758789039","path":"D:\\filebeatall\\test\\untiltest.txt","state-id":"native::65536-6772-2758789039","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"processors","log.origin":{"file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2023-07-25T06:43:47.003Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.8.2\"\n  },\n  \"event\": {\n    \"timezone\": \"+08:00\"\n  },\n  \"log\": {\n    \"offset\": 0,\n    \"file\": {\n      \"path\": \"D:\\\\filebeatall\\\\test\\\\untiltest.txt\"\n    }\n  },\n  \"message\": \"\\\"资源文件\\\"\",\n  \"input\": {\n    \"type\": \"filestream\"\n  },\n  \"fields\": {\n    \"logtopic\": \"test-mxl-filebeat123\"\n  },\n  \"host\": {\n    \"ip\": [\n      \"fe80::789d:6379:2985:47e2\",\n      \"172.24.131.126\",\n      \"fe80::5efe:ac18:837e\"\n    ],\n    \"mac\": [\n      \"00-00-00-00-00-00-00-E0\",\n      \"02-00-00-01-EF-14\"\n    ],\n    \"name\": \"KVM-XXXXXX\",\n    \"public_ip\": \"220.250.27.220\",\n    \"hostname\": \"KVM-XXXXXX\",\n    \"architecture\": \"x86_64\",\n    \"os\": {\n      \"name\": \"Windows Server 2008 R2 Enterprise\",\n      \"kernel\": \"6.1.7601.24545 (win7sp1_ldr_escrow.200102-1707)\",\n      \"build\": \"7601.24544\",\n      \"type\": \"windows\",\n      \"platform\": \"windows\",\n      \"version\": \"6.1\",\n      \"family\": \"windows\"\n    },\n    \"id\": \"cad5361a-c37e-48c6-b4b6-a9ce7713c726\"\n  },\n  \"agent\": {\n    \"type\": \"filebeat\",\n    \"version\": \"8.8.2\",\n    \"ephemeral_id\": \"92c14441-daa4-4c62-98f4-af2a357c3df6\",\n    \"id\": \"210d49b3-7f9c-4f39-a476-ed01f06c67da\",\n    \"name\": \"KVM-XXXXXX\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"processors","log.origin":{"file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n  \"@timestamp\": \"2023-07-25T06:43:47.003Z\",\n  \"@metadata\": {\n    \"beat\": \"filebeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.8.2\"\n  },\n  \"event\": {\n    \"timezone\": \"+08:00\"\n  },\n  \"log\": {\n    \"file\": {\n      \"path\": \"D:\\\\filebeatall\\\\test\\\\untiltest.txt\"\n    },\n    \"offset\": 18\n  },\n  \"message\": \"\\\"$_15_/wechatsetup/wechat.exe\\\"\",\n  \"fields\": {\n    \"logtopic\": \"test-mxl-filebeat123\"\n  },\n  \"input\": {\n    \"type\": \"filestream\"\n  },\n  \"host\": {\n    \"os\": {\n      \"type\": \"windows\",\n      \"platform\": \"windows\",\n      \"version\": \"6.1\",\n      \"family\": \"windows\",\n      \"name\": \"Windows Server 2008 R2 Enterprise\",\n      \"kernel\": \"6.1.7601.24545 (win7sp1_ldr_escrow.200102-1707)\",\n      \"build\": \"7601.24544\"\n    },\n    \"id\": \"cad5361a-c37e-48c6-b4b6-a9ce7713c726\",\n    \"name\": \"KVM-XXXXXX\",\n    \"public_ip\": \"220.250.27.220\",\n    \"ip\": [\n      \"fe80::789d:6379:2985:47e2\",\n      \"172.24.131.126\",\n      \"fe80::5efe:ac18:837e\"\n    ],\n    \"mac\": [\n      \"00-00-00-00-00-00-00-E0\",\n      \"02-00-00-01-EF-14\"\n    ],\n    \"hostname\": \"KVM-XXXXXX\",\n    \"architecture\": \"x86_64\"\n  },\n  \"agent\": {\n    \"ephemeral_id\": \"92c14441-daa4-4c62-98f4-af2a357c3df6\",\n    \"id\": \"210d49b3-7f9c-4f39-a476-ed01f06c67da\",\n    \"name\": \"KVM-XXXXXX\",\n    \"type\": \"filebeat\",\n    \"version\": \"8.8.2\"\n  },\n  \"ecs\": {\n    \"version\": \"8.0.0\"\n  }\n}","service.name":"filebeat","ecs.version":"1.6.0"}

Line 25 shows that after started in server B, the filebeat had loaded the filebeat registry files:

{"log.level":"info","@timestamp":"2023-07-25T14:43:44.966+0800","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for 'D:\\filebeatall\\filebeat-id\\data\\registry\\filebeat'. Active transaction id=16","service.name":"filebeat","ecs.version":"1.6.0"}

So, As we expect, It should resume harvest data rather than harvest from beginning.

6

Yeah, sorry, I didn't pay enough attention to what you are trying to do.

I don't think this will work, filebeat relies on the inode of the file, if you move the file to another server the inode will change and filebeat will read it again as it will be seem as a new file.

Even if you copy the registry file from one server to another, the inode of the file will be different.

1 Like
8

Thanks again!
Or we can change the file_identity to path ? We don't have the rotate log to harvest, so maybe it is the solution?
Log input | Filebeat Reference [7.15] | Elastic

9

It may work, but I think you will need to change it in both servers and this will also trigger a duplication if I'm not wrong.

You will need to test that.

10

I think maybe this issue is solved by adding configuration file_identity.path: ~.

- type: log
  enabled: true
  max_bytes: 204800
  encoding: utf-8
  paths:
    - D:\filebeatall\test\*.txt
  file_identity.path: ~
  fields:
    logtopic: test-lql-filebeat715

Although after server migration, it still has about 2048 lines duplicate, but it is acceptable.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.