# Data duplication problem after server migration
**URL:** https://discuss.elastic.co/t/data-duplication-problem-after-server-migration/339186
**Category:** Beats
**Tags:** filebeat
**Created:** [July 25, 2023, 12:03pm UTC](https://discuss.elastic.co/t/data-duplication-problem-after-server-migration/339186 "2023-07-25T12:03:36Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@charlielin](https://discuss.elastic.co/u/charlielin)
#### Post date: [July 25, 2023, 12:03pm UTC](https://discuss.elastic.co/t/data-duplication-problem-after-server-migration/339186/1 "2023-07-25T12:03:36Z")
Hi There:
Currently, we are using Filebeat (version 7.15.2) to harvest logs of program A and send them to Kafka. Due to some reason, we will do some operations called **`Server Migration`** freqently.
> **Server Migration** means:
>
> 1. Program A and Filebeat run on server A.
> 2. Stop Filebeat and program A on server A, copy all the logs and configurations of Program A, \> Filebeat, Filebeat data, Filebeat config, and Filebeat logs to server B, ensuring that the file's last modification time remains unchanged.
> 3. Restart Filebeat and Program A on server B.
After the server migration, we observed that Filebeat harvest logs which have been harvested on server A. After reading filebeat logs, we found that this issue may be related to the different input\_id which Filebeat generated on server A and server B separately.
Is there any solution to avoid this data duplication problem after server migration?
---
### Author:  [@charlielin](https://discuss.elastic.co/u/charlielin)
#### Post date: [July 25, 2023, 12:10pm UTC](https://discuss.elastic.co/t/data-duplication-problem-after-server-migration/339186/2 "2023-07-25T12:10:36Z")
Here is the filebeat log.json
```auto
{"op":"set","id":30}
{"k":"filebeat::logs::native::196608-45870-3781051510","v":{"prev_id":"","source":"D:\\filebeatall\\test\\untiltest.txt","offset":2026973,"timestamp":[2061775969080,1690194390],"ttl":-1,"type":"log","id":"native::196608-45870-3781051510","FileStateOS":{"idxhi":196608,"idxlo":45870,"vol":3781051510},"identifier_name":"native"}}
{"op":"set","id":31}
{"k":"filebeat::logs::native::196608-45870-3781051510","v":{"offset":2026973,"timestamp":[2061775969080,1690194390],"type":"log","prev_id":"","source":"D:\\filebeatall\\test\\untiltest.txt","ttl":-1,"FileStateOS":{"idxhi":196608,"idxlo":45870,"vol":3781051510},"identifier_name":"native","id":"native::196608-45870-3781051510"}}
{"op":"set","id":32}
{"k":"filebeat::logs::native::196608-45870-3781051510","v":{"source":"D:\\filebeatall\\test\\untiltest.txt","offset":2026973,"ttl":-1,"identifier_name":"native","prev_id":"","timestamp":[2061668698380,1690194516],"type":"log","FileStateOS":{"idxhi":196608,"idxlo":45870,"vol":3781051510},"id":"native::196608-45870-3781051510"}}
{"op":"set","id":33}
{"k":"filebeat::logs::native::196608-45870-3781051510","v":{"prev_id":"","source":"D:\\filebeatall\\test\\untiltest.txt","offset":2026973,"ttl":-1,"type":"log","id":"native::196608-45870-3781051510","timestamp":[2061668698380,1690194516],"FileStateOS":{"idxhi":196608,"idxlo":45870,"vol":3781051510},"identifier_name":"native"}}
{"op":"set","id":34}
{"k":"filebeat::logs::native::65536-1755945-118598","v":{"timestamp":[2061669697280,1690194516],"ttl":-1,"type":"log","identifier_name":"native","id":"native::65536-1755945-118598","prev_id":"","offset":0,"source":"D:\\filebeatall\\test\\untiltest.txt","FileStateOS":{"idxhi":65536,"idxlo":1755945,"vol":118598}}}
{"op":"remove","id":35}
{"k":"filebeat::logs::native::196608-45870-3781051510"}
{"op":"set","id":36}
{"k":"filebeat::logs::native::65536-1755945-118598","v":{"prev_id":"","offset":0,"FileStateOS":{"vol":118598,"idxhi":65536,"idxlo":1755945},"identifier_name":"native","id":"native::65536-1755945-118598","source":"D:\\filebeatall\\test\\untiltest.txt","timestamp":[2061669697280,1690194516],"ttl":-1,"type":"log"}}
{"op":"set","id":37}
{"k":"filebeat::logs::native::65536-1755945-118598","v":{"prev_id":"","source":"D:\\filebeatall\\test\\untiltest.txt","type":"log","FileStateOS":{"idxhi":65536,"idxlo":1755945,"vol":118598},"id":"native::65536-1755945-118598","offset":81453,"timestamp":[2061823930080,1690194564],"ttl":-1,"identifier_name":"native"}}
{"op":"set","id":38}
{"k":"filebeat::logs::native::65536-1755945-118598","v":{"prev_id":"","source":"D:\\filebeatall\\test\\untiltest.txt","offset":180721,"ttl":-1,"type":"log","id":"native::65536-1755945-118598","timestamp":[2062120246480,1690194564],"FileStateOS":{"idxhi":65536,"idxlo":1755945,"vol":118598},"identifier_name":"native"}}
{"op":"set","id":39}
{"k":"filebeat::logs::native::65536-1755945-118598","v":{"source":"D:\\filebeatall\\test\\untiltest.txt","offset":301156,"timestamp":[2062312176280,1690194564],"FileStateOS":{"idxhi":65536,"idxlo":1755945,"vol":118598},"identifier_name":"native","id":"native::65536-1755945-118598","prev_id":"","ttl":-1,"type":"log"}}
{"op":"set","id":40}
{"k":"filebeat::logs::native::65536-1755945-118598","v":{"identifier_name":"native","id":"native::65536-1755945-118598","prev_id":"","offset":393316,"type":"log","FileStateOS":{"idxhi":65536,"idxlo":1755945,"vol":118598},"source":"D:\\filebeatall\\test\\untiltest.txt","timestamp":[2062530576380,1690194564],"ttl":-1}}
{"op":"set","id":41}
{"k":"filebeat::logs::native::65536-1755945-118598","v":{"type":"log","identifier_name":"native","source":"D:\\filebeatall\\test\\untiltest.txt","offset":485476,"timestamp":[2061814945280,1690194565],"ttl":-1,"FileStateOS":{"idxhi":65536,"idxlo":1755945,"vol":118598},"id":"native::65536-1755945-118598","prev_id":""}}
```
The k changed from `filebeat::logs::native::196608-45870-3781051510` to `filebeat::logs::native::65536-1755945-118598`, and offset changed from `2026973` to `0` after server migration.
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [July 25, 2023, 12:20pm UTC](https://discuss.elastic.co/t/data-duplication-problem-after-server-migration/339186/3 "2023-07-25T12:20:40Z")
> [@charlielin](#):
>
> After reading filebeat logs, we found that this issue may be related to the different input\_id which Filebeat generated on server A and server B separately.
> Is there any solution to avoid this data duplication problem after server migration?
Yes, this is in the [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#filebeat-input-filestream).
> Each filestream input must have a unique ID. Omitting or changing the filestream ID may cause data duplication. Without a unique ID, filestream is unable to correctly track the state of files.
You need to add an `id` to your input and use the same id in both servers.
---
### Author:  [@charlielin](https://discuss.elastic.co/u/charlielin)
#### Post date: [July 25, 2023, 3:31pm UTC](https://discuss.elastic.co/t/data-duplication-problem-after-server-migration/339186/4 "2023-07-25T15:31:32Z")
Thansk for your reply.
Unfortunately, I have read this doc and tried before, so even we update the filebeat to 8.8 and use the filestream input plugin with the same `id` in both servers. This issue still occur.
Here is the filebeat log.json:
```auto
{"op":"set","id":14}
{"k":"filestream::my-streamfile-id::native::196608-45870-3781051510","v":{"ttl":1800000000000,"updated":[2062139226580,1690267384],"cursor":{"offset":973528},"meta":{"source":"D:\\filebeatall\\test\\untiltest.txt","identifier_name":"native"}}}
{"op":"set","id":15}
{"k":"filestream::my-streamfile-id::native::196608-45870-3781051510","v":{"cursor":{"offset":1054602},"meta":{"source":"D:\\filebeatall\\test\\untiltest.txt","identifier_name":"native"},"ttl":1800000000000,"updated":[2062325581080,1690267384]}}
{"op":"set","id":16}
{"k":"filestream::my-streamfile-id::native::196608-45870-3781051510","v":{"cursor":{"offset":1135894},"meta":{"source":"D:\\filebeatall\\test\\untiltest.txt","identifier_name":"native"},"ttl":1800000000000,"updated":[2062491778080,1690267384]}}
{"op":"set","id":17}
{"k":"filestream::my-streamfile-id::native::65536-6772-2758789039","v":{"ttl":0,"updated":[281470681743360,18446744011573954816],"cursor":null,"meta":{"source":"D:\\filebeatall\\test\\untiltest.txt","identifier_name":"native"}}}
{"op":"set","id":18}
{"k":"filestream::my-streamfile-id::native::65536-6772-2758789039","v":{"ttl":1800000000000,"updated":[2061588208280,1690267427],"cursor":null,"meta":{"identifier_name":"native","source":"D:\\filebeatall\\test\\untiltest.txt"}}}
{"op":"set","id":19}
{"k":"filestream::my-streamfile-id::native::65536-6772-2758789039","v":{"ttl":1800000000000,"updated":[2061927075480,1690267427],"cursor":{"offset":81453},"meta":{"source":"D:\\filebeatall\\test\\untiltest.txt","identifier_name":"native"}}}
{"op":"set","id":20}
{"k":"filestream::my-streamfile-id::native::65536-6772-2758789039","v":{"ttl":1800000000000,"updated":[2062287427080,1690267427],"cursor":{"offset":180721},"meta":{"source":"D:\\filebeatall\\test\\untiltest.txt","identifier_name":"native"}}}
{"op":"set","id":21}
{"k":"filestream::my-streamfile-id::native::65536-6772-2758789039","v":{"updated":[2061761059880,1690267428],"cursor":{"offset":301156},"meta":{"source":"D:\\filebeatall\\test\\untiltest.txt","identifier_name":"native"},"ttl":1800000000000}}
```
As you can see, while `my-streamfile-id` is the same id in both servers, The k changed from `filestream::my-streamfile-id::native::196608-45870-3781051510` in server A to `filestream::my-streamfile-id::native::65536-6772-2758789039` in server B, the offset changed from `1135894` in server A to `0` in server B.
---
### Author:  [@charlielin](https://discuss.elastic.co/u/charlielin)
#### Post date: [July 25, 2023, 3:42pm UTC](https://discuss.elastic.co/t/data-duplication-problem-after-server-migration/339186/5 "2023-07-25T15:42:06Z")
For more information, here is the log that filebeat started in server B:
```auto
{"log.level":"info","@timestamp":"2023-07-25T14:43:43.598+0800","log.origin":{"file.name":"instance/beat.go","file.line":779},"message":"Home path: [D:\\filebeatall\\filebeat-id] Config path: [D:\\filebeatall\\filebeat-id] Data path: [D:\\filebeatall\\filebeat-id\\data] Logs path: [D:\\filebeatall\\filebeat-id\\logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:43.599+0800","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":870},"message":"Beat metadata path: D:\\filebeatall\\filebeat-id\\data\\meta.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:43.599+0800","log.origin":{"file.name":"instance/beat.go","file.line":787},"message":"Beat ID: 210d49b3-7f9c-4f39-a476-ed01f06c67da","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:43.605+0800","log.origin":{"file.name":"instance/beat.go","file.line":816},"message":"Set max procs limit: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:43.613+0800","log.logger":"processors","log.origin":{"file.name":"processors/processor.go","file.line":114},"message":"Generated new processors: add_fields={\"host\":{\"public_ip\":\"220.250.27.220\"}}, add_locale=[format=offset], add_host_metadata=[netinfo.enabled=[true], cache.ttl=[5m0s]]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:43.613+0800","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":96},"message":"Syscall filtering is only supported on Linux","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:43.618+0800","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1299},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"D:\\filebeatall\\filebeat-id","data":"D:\\filebeatall\\filebeat-id\\data","home":"D:\\filebeatall\\filebeat-id","logs":"D:\\filebeatall\\filebeat-id\\logs"},"type":"filebeat","uuid":"210d49b3-7f9c-4f39-a476-ed01f06c67da"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-07-25T14:43:43.619+0800","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1308},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"92c6b2370e46e549acda91b396f665a7e51e249c","libbeat":"8.8.2","time":"2023-06-23T19:04:45.000Z","version":"8.8.2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-07-25T14:43:43.619+0800","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1311},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"windows","arch":"amd64","max_procs":1,"version":"go1.19.10"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-07-25T14:43:43.623+0800","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1317},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-07-25T10:21:27+08:00","name":"KVM-XXXXXX","ip":["fe80::789d:6379:2985:47e2","172.24.131.126","::1","127.0.0.1","fe80::5efe:ac18:837e"],"kernel_version":"6.1.7601.24545 (win7sp1_ldr_escrow.200102-1707)","mac":["02:00:00:01:ef:14","00:00:00:00:00:00:00:e0"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows Server 2008 R2 Enterprise","version":"6.1","major":6,"minor":1,"patch":0,"build":"7601.24544"},"timezone":"CST","timezone_offset_sec":28800,"id":"cad5361a-c37e-48c6-b4b6-a9ce7713c726"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-07-25T14:43:43.624+0800","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1346},"message":"Process info","service.name":"filebeat","system_info":{"process":{"cwd":"D:\\filebeatall\\filebeat-id","exe":"D:\\filebeatall\\filebeat-id\\filebeat.exe","name":"filebeat.exe","pid":9108,"ppid":9052,"start_time":"2023-07-25T14:43:43.338+0800"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-07-25T14:43:43.624+0800","log.origin":{"file.name":"instance/beat.go","file.line":330},"message":"Setup Beat: filebeat; Version: 8.8.2","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:43.627+0800","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":358},"message":"Initializing output plugins","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:44.966+0800","log.logger":"kafka","log.origin":{"file.name":"kafka/kafka.go","file.line":48},"message":"initialize kafka output","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:44.966+0800","log.logger":"publisher","log.origin":{"file.name":"pipeline/consumer.go","file.line":110},"message":"start pipeline event consumer","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:44.966+0800","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":105},"message":"Beat name: KVM-XXXXXX","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:44.966+0800","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-07-25T14:43:44.966+0800","log.origin":{"file.name":"beater/filebeat.go","file.line":175},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:44.966+0800","log.origin":{"file.name":"instance/beat.go","file.line":516},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:44.966+0800","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":287},"message":"isFile(D:\\filebeatall\\filebeat-id\\data\\registry\\) -> false","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:44.966+0800","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":287},"message":"isFile() -> false","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:44.966+0800","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":280},"message":"isDir(D:\\filebeatall\\filebeat-id\\data\\registry\\filebeat) -> true","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:44.966+0800","log.logger":"test","log.origin":{"file.name":"registrar/migrate.go","file.line":287},"message":"isFile(D:\\filebeatall\\filebeat-id\\data\\registry\\filebeat\\meta.json) -> true","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:44.966+0800","log.logger":"registrar","log.origin":{"file.name":"registrar/migrate.go","file.line":82},"message":"Registry type '1' found","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:44.966+0800","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for 'D:\\filebeatall\\filebeat-id\\data\\registry\\filebeat'. Active transaction id=16","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:44.990+0800","log.logger":"service","log.origin":{"file.name":"service/service_windows.go","file.line":110},"message":"Windows is interactive: true","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:44.990+0800","log.logger":"publisher","log.origin":{"file.name":"pipeline/queue_reader.go","file.line":49},"message":"pipeline event consumer queue reader: start","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:44.991+0800","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":145},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:44.999+0800","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for 'D:\\filebeatall\\filebeat-id\\data\\registry\\filebeat'. Active transaction id=16","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-07-25T14:43:44.999+0800","log.origin":{"file.name":"beater/filebeat.go","file.line":307},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform request:append","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform request:delete","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform request:set","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform response:append","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform response:delete","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform response:set","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform pagination:append","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform pagination:delete","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/transform_registry.go","file.line":75},"message":"Register transform pagination:set","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":84},"message":"registering encoder 'application/json': returned error: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":87},"message":"registering encoder 'application/x-www-form-urlencoded': returned error: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":93},"message":"registering decoder 'application/json': returned error: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":96},"message":"registering decoder 'application/x-ndjson': returned error: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":99},"message":"registering decoder 'text/csv': returned error: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"httpjson.transforms","log.origin":{"file.name":"httpjson/encoding.go","file.line":102},"message":"registering decoder 'application/zip': returned error: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:45.000+0800","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:45.001+0800","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:45.001+0800","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":140},"message":"Starting Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:45.001+0800","log.origin":{"file.name":"cfgfile/reload.go","file.line":163},"message":"Config reloader started","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"cfgfile","log.origin":{"file.name":"cfgfile/reload.go","file.line":193},"message":"Scan for new config files","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"cfgfile","log.origin":{"file.name":"cfgfile/cfgfile.go","file.line":204},"message":"Load config from file: D:\\filebeatall\\filebeat-id/input.yml","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"cfgfile","log.origin":{"file.name":"cfgfile/reload.go","file.line":212},"message":"Number of module configs found: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":90},"message":"Starting reload procedure, current runners: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":108},"message":"Start list: 1, Stop list: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"cfgwarn","log.origin":{"file.name":"cfgtype/byte_size.go","file.line":36},"message":"DEPRECATED: size now requires a unit (KiB, MiB, etc...), current value: 204800. Will be removed in version: 7.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":286},"message":"recursive glob enabled","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":141},"message":"Starting runner: filestream","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":120},"message":"Input 'filestream' starting","service.name":"filebeat","id":"my-streamfile-id","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"input.filestream","log.origin":{"file.name":"filestream/prospector.go","file.line":131},"message":"Starting prospector","service.name":"filebeat","id":"my-streamfile-id","prospector":"file_prospector","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":139},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":213},"message":"Found 1 paths","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"input.filestream","log.origin":{"file.name":"filestream/prospector.go","file.line":177},"message":"A new file D:\\filebeatall\\test\\untiltest.txt has been found","service.name":"filebeat","id":"my-streamfile-id","prospector":"file_prospector","operation":"create","source_name":"native::65536-6772-2758789039","os_id":"65536-6772-2758789039","new_path":"D:\\filebeatall\\test\\untiltest.txt","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"input.filestream","log.origin":{"file.name":"input-logfile/harvester.go","file.line":138},"message":"Starting harvester for file","service.name":"filebeat","id":"my-streamfile-id","source_file":"filestream::my-streamfile-id::native::65536-6772-2758789039","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":176},"message":"newLogFileReader with config.MaxBytes:10485760","service.name":"filebeat","id":"my-streamfile-id","source_file":"filestream::my-streamfile-id::native::65536-6772-2758789039","path":"D:\\filebeatall\\test\\untiltest.txt","state-id":"native::65536-6772-2758789039","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"processors","log.origin":{"file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n \"@timestamp\": \"2023-07-25T06:43:47.003Z\",\n \"@metadata\": {\n \"beat\": \"filebeat\",\n \"type\": \"_doc\",\n \"version\": \"8.8.2\"\n },\n \"event\": {\n \"timezone\": \"+08:00\"\n },\n \"log\": {\n \"offset\": 0,\n \"file\": {\n \"path\": \"D:\\\\filebeatall\\\\test\\\\untiltest.txt\"\n }\n },\n \"message\": \"\\\"资源文件\\\"\",\n \"input\": {\n \"type\": \"filestream\"\n },\n \"fields\": {\n \"logtopic\": \"test-mxl-filebeat123\"\n },\n \"host\": {\n \"ip\": [\n \"fe80::789d:6379:2985:47e2\",\n \"172.24.131.126\",\n \"fe80::5efe:ac18:837e\"\n],\n \"mac\": [\n \"00-00-00-00-00-00-00-E0\",\n \"02-00-00-01-EF-14\"\n],\n \"name\": \"KVM-XXXXXX\",\n \"public_ip\": \"220.250.27.220\",\n \"hostname\": \"KVM-XXXXXX\",\n \"architecture\": \"x86_64\",\n \"os\": {\n \"name\": \"Windows Server 2008 R2 Enterprise\",\n \"kernel\": \"6.1.7601.24545 (win7sp1_ldr_escrow.200102-1707)\",\n \"build\": \"7601.24544\",\n \"type\": \"windows\",\n \"platform\": \"windows\",\n \"version\": \"6.1\",\n \"family\": \"windows\"\n },\n \"id\": \"cad5361a-c37e-48c6-b4b6-a9ce7713c726\"\n },\n \"agent\": {\n \"type\": \"filebeat\",\n \"version\": \"8.8.2\",\n \"ephemeral_id\": \"92c14441-daa4-4c62-98f4-af2a357c3df6\",\n \"id\": \"210d49b3-7f9c-4f39-a476-ed01f06c67da\",\n \"name\": \"KVM-XXXXXX\"\n },\n \"ecs\": {\n \"version\": \"8.0.0\"\n }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-07-25T14:43:47.003+0800","log.logger":"processors","log.origin":{"file.name":"processing/processors.go","file.line":213},"message":"Publish event: {\n \"@timestamp\": \"2023-07-25T06:43:47.003Z\",\n \"@metadata\": {\n \"beat\": \"filebeat\",\n \"type\": \"_doc\",\n \"version\": \"8.8.2\"\n },\n \"event\": {\n \"timezone\": \"+08:00\"\n },\n \"log\": {\n \"file\": {\n \"path\": \"D:\\\\filebeatall\\\\test\\\\untiltest.txt\"\n },\n \"offset\": 18\n },\n \"message\": \"\\\"$_15_/wechatsetup/wechat.exe\\\"\",\n \"fields\": {\n \"logtopic\": \"test-mxl-filebeat123\"\n },\n \"input\": {\n \"type\": \"filestream\"\n },\n \"host\": {\n \"os\": {\n \"type\": \"windows\",\n \"platform\": \"windows\",\n \"version\": \"6.1\",\n \"family\": \"windows\",\n \"name\": \"Windows Server 2008 R2 Enterprise\",\n \"kernel\": \"6.1.7601.24545 (win7sp1_ldr_escrow.200102-1707)\",\n \"build\": \"7601.24544\"\n },\n \"id\": \"cad5361a-c37e-48c6-b4b6-a9ce7713c726\",\n \"name\": \"KVM-XXXXXX\",\n \"public_ip\": \"220.250.27.220\",\n \"ip\": [\n \"fe80::789d:6379:2985:47e2\",\n \"172.24.131.126\",\n \"fe80::5efe:ac18:837e\"\n],\n \"mac\": [\n \"00-00-00-00-00-00-00-E0\",\n \"02-00-00-01-EF-14\"\n],\n \"hostname\": \"KVM-XXXXXX\",\n \"architecture\": \"x86_64\"\n },\n \"agent\": {\n \"ephemeral_id\": \"92c14441-daa4-4c62-98f4-af2a357c3df6\",\n \"id\": \"210d49b3-7f9c-4f39-a476-ed01f06c67da\",\n \"name\": \"KVM-XXXXXX\",\n \"type\": \"filebeat\",\n \"version\": \"8.8.2\"\n },\n \"ecs\": {\n \"version\": \"8.0.0\"\n }\n}","service.name":"filebeat","ecs.version":"1.6.0"}
```
Line 25 shows that after started in server B, the filebeat had loaded the filebeat registry files:
```auto
{"log.level":"info","@timestamp":"2023-07-25T14:43:44.966+0800","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for 'D:\\filebeatall\\filebeat-id\\data\\registry\\filebeat'. Active transaction id=16","service.name":"filebeat","ecs.version":"1.6.0"}
```
So, As we expect, It should resume harvest data rather than harvest from beginning.
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [July 25, 2023, 3:47pm UTC](https://discuss.elastic.co/t/data-duplication-problem-after-server-migration/339186/6 "2023-07-25T15:47:30Z")
Yeah, sorry, I didn't pay enough attention to what you are trying to do.
> [@charlielin](#):
>
> - Program A and Filebeat run on server A.
> - Stop Filebeat and program A on server A, copy all the logs and configurations of Program A, \> Filebeat, Filebeat data, Filebeat config, and Filebeat logs to server B, ensuring that the file's last modification time remains unchanged.
> - Restart Filebeat and Program A on server B.
I don't think this will work, filebeat relies on the **inode** of the file, if you move the file to another server the inode will change and filebeat will read it again as it will be seem as a new file.
Even if you copy the registry file from one server to another, the inode of the file will be different.
---
### Author:  [@charlielin](https://discuss.elastic.co/u/charlielin)
#### Post date: [July 25, 2023, 4:17pm UTC](https://discuss.elastic.co/t/data-duplication-problem-after-server-migration/339186/8 "2023-07-25T16:17:00Z")
Thanks again!
Or we can change the `file_identity` to `path` ? We don't have the rotate log to harvest, so maybe it is the solution?
[Log input | Filebeat Reference [7.15] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/7.15/filebeat-input-log.html)
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [July 25, 2023, 4:20pm UTC](https://discuss.elastic.co/t/data-duplication-problem-after-server-migration/339186/9 "2023-07-25T16:20:00Z")
It may work, but I think you will need to change it in both servers and this will also trigger a duplication if I'm not wrong.
You will need to test that.
---
### Author:  [@charlielin](https://discuss.elastic.co/u/charlielin)
#### Post date: [July 26, 2023, 4:41pm UTC](https://discuss.elastic.co/t/data-duplication-problem-after-server-migration/339186/10 "2023-07-26T16:41:44Z")
I think maybe this issue is solved by adding configuration `file_identity.path: ~`.
```yaml
- type: log
enabled: true
max_bytes: 204800
encoding: utf-8
paths:
- D:\filebeatall\test\*.txt
file_identity.path: ~
fields:
logtopic: test-lql-filebeat715
```
Although after server migration, it still has about 2048 lines duplicate, but it is acceptable.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 23, 2023, 6:42pm UTC](https://discuss.elastic.co/t/data-duplication-problem-after-server-migration/339186/11 "2023-08-23T18:42:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.