# After filebeat upgrades & preserving registry it sending duplicate events
**URL:** https://discuss.elastic.co/t/after-filebeat-upgrades-preserving-registry-it-sending-duplicate-events/313701
**Category:** Beats
**Tags:** filebeat
**Created:** [September 5, 2022, 3:56pm UTC](https://discuss.elastic.co/t/after-filebeat-upgrades-preserving-registry-it-sending-duplicate-events/313701 "2022-09-05T15:56:13Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@krish0608](https://discuss.elastic.co/u/krish0608)
#### Post date: [September 5, 2022, 3:56pm UTC](https://discuss.elastic.co/t/after-filebeat-upgrades-preserving-registry-it-sending-duplicate-events/313701/1 "2022-09-05T15:56:13Z")
I am having a trouble with duplicate events in elasticsearch after updating it to 7.17.3 from 5.6.5. In case of Linux its working as expected but in windows its sending duplicate events. Below are my configurations:
========filebeat.yml========\>
```auto
filebeat.inputs:
- type: filestream
filebeat.config.inputs:
path: C:/opt/filebeat-7.17.3-windows-x86_64/conf.d
# ------------------------------ Logstash Output -------------------------------
output.logstash:
# The Logstash hosts
hosts: ["do-test:12203"]
# ================================== Logging ===================================
# Sets log level. The default log level is info.
logging:
level: debug
to_files: true
files:
rotateeverybytes: 10485760
keepfiles: 7
```
========Extra Configuration============\>
```auto
- paths:
- D:\wsod\Logs\iis\*\*
exclude_files:
- HTTPERR
exclude_lines:
- "^#"
- "\\/monitor\\/wsod.asp - \\d+ [\\d\\.]+ - 200"
ignore_older: 25h
fields:
type: iis
environment: development
datacenter: ptc1
tags:
- iis
- webdev
fields_under_root: true
```
In Linux, Registry is working as expected but in case of windows its sending duplicate events not sure what configuration am doing wrong. Please help me out.
@steffens
---
### Author:  [@krish0608](https://discuss.elastic.co/u/krish0608)
#### Post date: [September 7, 2022, 1:46pm UTC](https://discuss.elastic.co/t/after-filebeat-upgrades-preserving-registry-it-sending-duplicate-events/313701/2 "2022-09-07T13:46:08Z")
Can anyone please help me out??
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [September 7, 2022, 2:23pm UTC](https://discuss.elastic.co/t/after-filebeat-upgrades-preserving-registry-it-sending-duplicate-events/313701/4 "2022-09-07T14:23:32Z")
Hi @krish0608
It is not clear what your problem is you have not provide enough information.
That was a _very_ long time between upgrades 7.17.3 from 5.6.5. so it is hard to know what the issues is.
How many duplicate events... just some or every event?
Are you receiving duplicate events ongoing?
or just when you start it the first time after the upgrade?
Is there any errors in the filebeat logs?
Are the permission correct on the filebeat data directory?
---
### Author:  [@krish0608](https://discuss.elastic.co/u/krish0608)
#### Post date: [September 13, 2022, 12:55pm UTC](https://discuss.elastic.co/t/after-filebeat-upgrades-preserving-registry-it-sending-duplicate-events/313701/5 "2022-09-13T12:55:38Z")
Hi @stephenb
Thanks for your revert. Please find the answers to your questions below:-
1. How many duplicate events:- its sending current date log file again.
2. Are you receiving duplicate events ongoing?- The process i followed was i checked no of Logs count in Kibana before upgrade and after upgrade it resend current date events from Logfile. And yes it the first time after the upgrade.
3. Yes Permission were correct.
Just FYI, We are facing challenges in case of windows, for Linux its working fine. Also regestry preservation is working as expected in both case i have checked offset value after upgrade it was present in new registry file.
Please find the attached filebeat Logs:-
```auto
2022-09-13T08:34:15.357-0400 INFO instance/beat.go:685 Home path: [c:\program files\filebeat] Config path: [c:\program files\filebeat] Data path: [c:\program files\filebeat\data] Logs path: [D:\wsod\logs\filebeat] Hostfs Path: [/]
2022-09-13T08:34:15.365-0400 INFO instance/beat.go:693 Beat ID: 2906c257-019f-4da2-b6a7-450ff28165fc
2022-09-13T08:34:15.366-0400 INFO [beat] instance/beat.go:1039 Beat info {"system_info": {"beat": {"path": {"config": "c:\\program files\\filebeat", "data": "c:\\program files\\filebeat\\data", "home": "c:\\program files\\filebeat", "logs": "D:\\wsod\\logs\\filebeat"}, "type": "filebeat", "uuid": "2906c257-019f-4da2-b6a7-450ff28165fc"}}}
2022-09-13T08:34:15.366-0400 INFO [beat] instance/beat.go:1048 Build info {"system_info": {"build": {"commit": "1993ee88a11cb34f61a1fb45c7c3cf50533682cb", "libbeat": "7.17.3", "time": "2022-04-19T09:27:20.000Z", "version": "7.17.3"}}}
2022-09-13T08:34:15.366-0400 INFO [beat] instance/beat.go:1051 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":2,"version":"go1.17.8"}}}
2022-09-13T08:34:15.371-0400 INFO [beat] instance/beat.go:1055 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-08-13T18:36:19.28-04:00","name":"MD-WWWD702A","ip":["fe80::ad8a:626:83ab:1cd5/64","10.124.0.81/23","::1/128","127.0.0.1/8","fe80::5efe:a7c:51/128"],"kernel_version":"6.3.9600.20512 (winblue_ltsb_escrow.220711-1746)","mac":["00:50:56:ab:32:4c","00:00:00:00:00:00:00:e0"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows Server 2012 R2 Standard","version":"6.3","major":3,"minor":0,"patch":0,"build":"9600.20520"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"871f492a-4abc-4714-9af1-0171c375f116"}}}
2022-09-13T08:34:15.371-0400 INFO [beat] instance/beat.go:1084 Process info {"system_info": {"process": {"cwd": "C:\\Windows\\system32", "exe": "C:\\Program Files\\filebeat\\filebeat.exe", "name": "filebeat.exe", "pid": 6408, "ppid": 628, "start_time": "2022-09-13T08:34:14.152-0400"}}}
2022-09-13T08:34:15.371-0400 INFO instance/beat.go:328 Setup Beat: filebeat; Version: 7.17.3
2022-09-13T08:34:15.372-0400 INFO [publisher] pipeline/module.go:113 Beat name: MD-WWWD702A
2022-09-13T08:34:15.372-0400 WARN beater/filebeat.go:202 Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.
2022-09-13T08:34:15.373-0400 INFO instance/beat.go:492 filebeat start running.
2022-09-13T08:34:15.373-0400 INFO registrar/migrate.go:129 Migrate registry file to registry directory
2022-09-13T08:34:15.373-0400 INFO registrar/migrate.go:134 Move registry file to backup file: C:\\Program Files\\filebeat\\data\\registry.bak
2022-09-13T08:34:15.373-0400 INFO registrar/migrate.go:163 No registry home found. Create: C:\Program Files\filebeat\data\registry\filebeat
2022-09-13T08:34:15.374-0400 INFO registrar/migrate.go:171 Initialize registry meta file
2022-09-13T08:34:15.375-0400 INFO [monitoring] log/log.go:142 Starting metrics logging every 30s
2022-09-13T08:34:15.376-0400 INFO registrar/migrate.go:151 Migrate old registry file to new data file
2022-09-13T08:34:15.376-0400 INFO registrar/migrate.go:185 Migrate registry version 0 to version 1
2022-09-13T08:34:15.377-0400 INFO memlog/store.go:119 Loading data file of 'C:\Program Files\filebeat\data\registry\filebeat' succeeded. Active transaction id=0
2022-09-13T08:34:15.377-0400 INFO memlog/store.go:124 Finished loading transaction log file for 'C:\Program Files\filebeat\data\registry\filebeat'. Active transaction id=0
2022-09-13T08:34:15.384-0400 INFO memlog/store.go:119 Loading data file of 'C:\Program Files\filebeat\data\registry\filebeat' succeeded. Active transaction id=4
2022-09-13T08:34:15.384-0400 INFO memlog/store.go:124 Finished loading transaction log file for 'C:\Program Files\filebeat\data\registry\filebeat'. Active transaction id=4
2022-09-13T08:34:15.384-0400 WARN beater/filebeat.go:411 Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.
2022-09-13T08:34:15.385-0400 INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 3
2022-09-13T08:34:15.385-0400 INFO [crawler] beater/crawler.go:71 Loading Inputs: 1
2022-09-13T08:34:15.385-0400 INFO [crawler] beater/crawler.go:117 starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.type]
2022-09-13T08:34:15.385-0400 INFO [crawler] beater/crawler.go:121 input disabled, skipping it
2022-09-13T08:34:15.386-0400 WARN [cfgwarn] log/input.go:89 DEPRECATED: Log input. Use Filestream input instead.
2022-09-13T08:34:15.386-0400 INFO [input] log/input.go:171 Configured paths: [D:\wsod\Logs\iis\httperr\*.log] {"input_id": "5705bceb-9bde-40a1-94f5-8b4fcfd31857"}
2022-09-13T08:34:15.386-0400 INFO [input] log/input.go:171 Configured paths: [D:\wsod\Logs\iis\*\*.log] {"input_id": "fff6014b-db7e-45cf-bd41-0b5d5b9bbb5c"}
2022-09-13T08:34:15.387-0400 INFO [input] log/input.go:171 Configured paths: [D:\wsod\Logs\web\*\*.log] {"input_id": "e6232215-2907-4203-975d-66b9ae9c60ae"}
2022-09-13T08:34:15.387-0400 INFO [crawler] beater/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 0
2022-09-13T08:34:15.387-0400 INFO cfgfile/reload.go:164 Config reloader started
2022-09-13T08:34:15.388-0400 INFO [input] log/input.go:171 Configured paths: [D:\wsod\Logs\iis\httperr\*.log] {"input_id": "311a10dd-85d3-4a00-84f3-fe27a78a01a1"}
2022-09-13T08:34:15.388-0400 INFO [input] log/input.go:171 Configured paths: [D:\wsod\Logs\iis\*\*.log] {"input_id": "dc698c8a-cd69-4039-b7c3-773b19e41b18"}
2022-09-13T08:34:15.389-0400 INFO [input] log/input.go:171 Configured paths: [D:\wsod\Logs\web\*\*.log] {"input_id": "9d261f77-31fc-4d79-a475-9f31835d08b6"}
2022-09-13T08:34:15.389-0400 INFO cfgfile/reload.go:224 Loading of config files completed.
2022-09-13T08:34:15.389-0400 INFO [input.harvester] log/harvester.go:309 Harvester started for paths: [D:\wsod\Logs\iis\httperr\*.log] {"input_id": "311a10dd-85d3-4a00-84f3-fe27a78a01a1", "source": "D:\\wsod\\Logs\\iis\\httperr\\httperr10587.log", "state_id": "native::475398144-56586-2282207034", "finished": false, "os_id": "475398144-56586-2282207034", "harvester_id": "ab30fbe2-13d3-421f-9045-47caf2e56f31"}
2022-09-13T08:34:15.391-0400 INFO [input.harvester] log/harvester.go:309 Harvester started for paths: [D:\wsod\Logs\iis\*\*.log] {"input_id": "dc698c8a-cd69-4039-b7c3-773b19e41b18", "source": "D:\\wsod\\Logs\\iis\\W3SVC12345\\u_ex220913.log", "state_id": "native::201916416-65048-2282207034", "finished": false, "os_id": "201916416-65048-2282207034", "harvester_id": "9563af3a-5edf-4541-a5bf-76ae5fbb07df"}
2022-09-13T08:34:15.391-0400 INFO [input.harvester] log/harvester.go:309 Harvester started for paths: [D:\wsod\Logs\iis\*\*.log] {"input_id": "dc698c8a-cd69-4039-b7c3-773b19e41b18", "source": "D:\\wsod\\Logs\\iis\\W3SVC12345\\u_ex220912.log", "state_id": "native::192806912-65201-2282207034", "finished": false, "os_id": "192806912-65201-2282207034", "harvester_id": "02e2ec64-18d8-4667-af17-301afb68efc6"}
2022-09-13T08:34:15.439-0400 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to failover(backoff(async(tcp://do-logd701.dev1.mdgapp.net:12203)),backoff(async(tcp://do-logd702.dev1.mdgapp.net:12203)))
2022-09-13T08:34:15.447-0400 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2022-09-13T08:34:15.447-0400 INFO [publisher] pipeline/retry.go:223 done
2022-09-13T08:34:15.470-0400 INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to failover(backoff(async(tcp://do-logd701.dev1.mdgapp.net:12203)),backoff(async(tcp://do-logd702.dev1.mdgapp.net:12203))) established
2022-09-13T08:34:45.391-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1109,"time":{"ms":1109}},"total":{"ticks":8671,"time":{"ms":8671},"value":8671},"user":{"ticks":7562,"time":{"ms":7562}}},"handles":{"open":167},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":30498},"version":"7.17.3"},"memstats":{"gc_next":32619232,"memory_alloc":18205648,"memory_sys":49340440,"memory_total":1255245816,"rss":72171520},"runtime":{"goroutines":59}},"filebeat":{"events":{"added":279429,"done":279429},"harvester":{"open_files":3,"running":3,"started":3}},"libbeat":{"config":{"module":{"running":3,"starts":3},"reloads":1,"scans":1},"output":{"events":{"acked":271451,"active":0,"batches":133,"total":271451},"read":{"bytes":798},"type":"logstash","write":{"bytes":10626131}},"pipeline":{"clients":3,"events":{"active":0,"filtered":7978,"published":271451,"retry":2048,"total":279429},"queue":{"acked":271451,"max_events":4096}}},"registrar":{"states":{"current":3,"update":279429},"writes":{"success":161,"total":161}},"system":{"cpu":{"cores":2}}}}}
2022-09-13T08:35:15.392-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1125,"time":{"ms":16}},"total":{"ticks":8703,"time":{"ms":32},"value":8703},"user":{"ticks":7578,"time":{"ms":16}}},"handles":{"open":167},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":60502},"version":"7.17.3"},"memstats":{"gc_next":32619232,"memory_alloc":19969560,"memory_total":1257009728,"rss":72663040},"runtime":{"goroutines":59}},"filebeat":{"events":{"added":52,"done":52},"harvester":{"open_files":3,"running":3}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":52,"active":0,"batches":1,"total":52},"read":{"bytes":6},"write":{"bytes":2581}},"pipeline":{"clients":3,"events":{"active":0,"published":52,"total":52},"queue":{"acked":52}}},"registrar":{"states":{"current":3,"update":52},"writes":{"success":1,"total":1}}}}}
2022-09-13T08:35:45.409-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1125},"total":{"ticks":8703,"value":8703},"user":{"ticks":7578}},"handles":{"open":167},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":90511},"version":"7.17.3"},"memstats":{"gc_next":32619232,"memory_alloc":20215232,"memory_total":1257255400,"rss":72683520},"runtime":{"goroutines":59}},"filebeat":{"harvester":{"open_files":3,"running":3}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"active":0}},"pipeline":{"clients":3,"events":{"active":0}}},"registrar":{"states":{"current":3}}}}}
2022-09-13T08:36:15.411-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1125},"total":{"ticks":8703,"value":8703},"user":{"ticks":7578}},"handles":{"open":169},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":120516},"version":"7.17.3"},"memstats":{"gc_next":32619232,"memory_alloc":22084568,"memory_total":1259124736,"rss":73654272},"runtime":{"goroutines":59}},"filebeat":{"events":{"added":86,"done":86},"harvester":{"open_files":3,"running":3}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":86,"active":0,"batches":1,"total":86},"read":{"bytes":6},"write":{"bytes":3876}},"pipeline":{"clients":3,"events":{"active":0,"published":86,"total":86},"queue":{"acked":86}}},"registrar":{"states":{"current":3,"update":86},"writes":{"success":1,"total":1}}}}}
2022-09-13T08:36:45.400-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1125},"total":{"ticks":8703,"value":8703},"user":{"ticks":7578}},"handles":{"open":169},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":150496},"version":"7.17.3"},"memstats":{"gc_next":21038528,"memory_alloc":10519256,"memory_total":1259371880,"rss":57937920},"runtime":{"goroutines":59}},"filebeat":{"harvester":{"open_files":3,"running":3}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"active":0}},"pipeline":{"clients":3,"events":{"active":0}}},"registrar":{"states":{"current":3}}}}}
2022-09-13T08:37:15.409-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1125,"time":{"ms":15}},"total":{"ticks":8703,"time":{"ms":15},"value":8703},"user":{"ticks":7578}},"handles":{"open":165},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":180504},"version":"7.17.3"},"memstats":{"gc_next":21038528,"memory_alloc":12436504,"memory_total":1261289128,"rss":57913344},"runtime":{"goroutines":59}},"filebeat":{"events":{"added":121,"done":121},"harvester":{"open_files":3,"running":3}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":121,"active":0,"batches":1,"total":121},"read":{"bytes":6},"write":{"bytes":5335}},"pipeline":{"clients":3,"events":{"active":0,"published":121,"total":121},"queue":{"acked":121}}},"registrar":{"states":{"current":3,"update":121},"writes":{"success":1,"total":1}}}}}
2022-09-13T08:37:45.410-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1140},"total":{"ticks":8718,"time":{"ms":15},"value":8718},"user":{"ticks":7578,"time":{"ms":15}}},"handles":{"open":167},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":210502},"version":"7.17.3"},"memstats":{"gc_next":21038528,"memory_alloc":12760944,"memory_total":1261613568,"rss":57933824},"runtime":{"goroutines":59}},"filebeat":{"harvester":{"open_files":3,"running":3}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"active":0}},"pipeline":{"clients":3,"events":{"active":0}}},"registrar":{"states":{"current":3}}}}}
2022-09-13T08:38:15.415-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1171,"time":{"ms":31}},"total":{"ticks":8764,"time":{"ms":31},"value":8764},"user":{"ticks":7593}},"handles":{"open":169},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":240508},"version":"7.17.3"},"memstats":{"gc_next":21038528,"memory_alloc":14508560,"memory_sys":262144,"memory_total":1263361184,"rss":57950208},"runtime":{"goroutines":59}},"filebeat":{"events":{"added":52,"done":52},"harvester":{"open_files":3,"running":3}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":52,"active":0,"batches":1,"total":52},"read":{"bytes":6},"write":{"bytes":2654}},"pipeline":{"clients":3,"events":{"active":0,"published":52,"total":52},"queue":{"acked":52}}},"registrar":{"states":{"current":3,"update":52},"writes":{"success":1,"total":1}}}}}
2022-09-13T08:38:45.419-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1203,"time":{"ms":32}},"total":{"ticks":8796,"time":{"ms":32},"value":8796},"user":{"ticks":7593}},"handles":{"open":167},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":270506},"version":"7.17.3"},"memstats":{"gc_next":21009840,"memory_alloc":10504712,"memory_total":1263606336,"rss":53972992},"runtime":{"goroutines":59}},"filebeat":{"harvester":{"open_files":3,"running":3}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"active":0}},"pipeline":{"clients":3,"events":{"active":0}}},"registrar":{"states":{"current":3}}}}}
2022-09-13T08:39:15.413-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1234,"time":{"ms":31}},"total":{"ticks":8827,"time":{"ms":31},"value":8827},"user":{"ticks":7593}},"handles":{"open":169},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":300497},"version":"7.17.3"},"memstats":{"gc_next":21009840,"memory_alloc":12219616,"memory_total":1265321240,"rss":54321152},"runtime":{"goroutines":59}},"filebeat":{"events":{"added":55,"done":55},"harvester":{"open_files":3,"running":3}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":55,"active":0,"batches":1,"total":55},"read":{"bytes":6},"write":{"bytes":2714}},"pipeline":{"clients":3,"events":{"active":0,"published":55,"total":55},"queue":{"acked":55}}},"registrar":{"states":{"current":3,"update":55},"writes":{"success":1,"total":1}}}}}
2022-09-13T08:39:20.638-0400 INFO [input.harvester] log/harvester.go:340 File is inactive. Closing because close_inactive of 5m0s reached. {"input_id": "311a10dd-85d3-4a00-84f3-fe27a78a01a1", "source": "D:\\wsod\\Logs\\iis\\httperr\\httperr10587.log", "state_id": "native::475398144-56586-2282207034", "finished": false, "os_id": "475398144-56586-2282207034", "harvester_id": "ab30fbe2-13d3-421f-9045-47caf2e56f31"}
2022-09-13T08:39:45.415-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1234},"total":{"ticks":8827,"value":8827},"user":{"ticks":7593}},"handles":{"open":168},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":330496},"version":"7.17.3"},"memstats":{"gc_next":21009840,"memory_alloc":12555032,"memory_total":1265656656,"rss":54325248},"runtime":{"goroutines":54}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"closed":1,"open_files":2,"running":2}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"active":0}},"pipeline":{"clients":3,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":3,"update":1},"writes":{"success":1,"total":1}}}}}
2022-09-13T08:39:45.638-0400 INFO [input.harvester] log/harvester.go:340 File is inactive. Closing because close_inactive of 5m0s reached. {"input_id": "dc698c8a-cd69-4039-b7c3-773b19e41b18", "source": "D:\\wsod\\Logs\\iis\\W3SVC12345\\u_ex220912.log", "state_id": "native::192806912-65201-2282207034", "finished": false, "os_id": "192806912-65201-2282207034", "harvester_id": "02e2ec64-18d8-4667-af17-301afb68efc6"}
2022-09-13T08:40:15.429-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1234,"time":{"ms":16}},"total":{"ticks":8843,"time":{"ms":32},"value":8843},"user":{"ticks":7609,"time":{"ms":16}}},"handles":{"open":165},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":360513},"version":"7.17.3"},"memstats":{"gc_next":21009840,"memory_alloc":14460872,"memory_total":1267562496,"rss":55193600},"runtime":{"goroutines":49}},"filebeat":{"events":{"added":93,"done":93},"harvester":{"closed":1,"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":92,"active":0,"batches":1,"total":92},"read":{"bytes":6},"write":{"bytes":4174}},"pipeline":{"clients":3,"events":{"active":0,"filtered":1,"published":92,"total":93},"queue":{"acked":92}}},"registrar":{"states":{"current":3,"update":93},"writes":{"success":2,"total":2}}}}}
2022-09-13T08:40:45.425-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1265,"time":{"ms":15}},"total":{"ticks":8874,"time":{"ms":15},"value":8874},"user":{"ticks":7609}},"handles":{"open":167},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":390505},"version":"7.17.3"},"memstats":{"gc_next":20932096,"memory_alloc":10546056,"memory_total":1267883256,"rss":53850112},"runtime":{"goroutines":49}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"active":0}},"pipeline":{"clients":3,"events":{"active":0}}},"registrar":{"states":{"current":3}}}}}
2022-09-13T08:41:15.433-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1265,"time":{"ms":16}},"total":{"ticks":8874,"time":{"ms":16},"value":8874},"user":{"ticks":7609}},"handles":{"open":167},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":420505},"version":"7.17.3"},"memstats":{"gc_next":20932096,"memory_alloc":12241280,"memory_total":1269578480,"rss":54198272},"runtime":{"goroutines":49}},"filebeat":{"events":{"added":72,"done":72},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":72,"active":0,"batches":1,"total":72},"read":{"bytes":6},"write":{"bytes":3335}},"pipeline":{"clients":3,"events":{"active":0,"published":72,"total":72},"queue":{"acked":72}}},"registrar":{"states":{"current":3,"update":72},"writes":{"success":1,"total":1}}}}}
2022-09-13T08:41:45.437-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1312,"time":{"ms":31}},"total":{"ticks":8921,"time":{"ms":31},"value":8921},"user":{"ticks":7609}},"handles":{"open":165},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":450504},"version":"7.17.3"},"memstats":{"gc_next":20932096,"memory_alloc":12562600,"memory_total":1269899800,"rss":54198272},"runtime":{"goroutines":49}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"active":0}},"pipeline":{"clients":3,"events":{"active":0}}},"registrar":{"states":{"current":3}}}}}
2022-09-13T08:42:15.429-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1343,"time":{"ms":31}},"total":{"ticks":8952,"time":{"ms":31},"value":8952},"user":{"ticks":7609}},"handles":{"open":167},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":480497},"version":"7.17.3"},"memstats":{"gc_next":20932096,"memory_alloc":14493696,"memory_total":1271830896,"rss":55070720},"runtime":{"goroutines":49}},"filebeat":{"events":{"added":114,"done":114},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":114,"active":0,"batches":1,"total":114},"read":{"bytes":6},"write":{"bytes":5056}},"pipeline":{"clients":3,"events":{"active":0,"published":114,"total":114},"queue":{"acked":114}}},"registrar":{"states":{"current":3,"update":114},"writes":{"success":1,"total":1}}}}}
2022-09-13T08:42:45.433-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1359,"time":{"ms":16}},"total":{"ticks":8968,"time":{"ms":16},"value":8968},"user":{"ticks":7609}},"handles":{"open":167},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":510497},"version":"7.17.3"},"memstats":{"gc_next":21075760,"memory_alloc":10538096,"memory_total":1272152472,"rss":54050816},"runtime":{"goroutines":49}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"active":0}},"pipeline":{"clients":3,"events":{"active":0}}},"registrar":{"states":{"current":3}}}}}
2022-09-13T08:43:15.438-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1390,"time":{"ms":31}},"total":{"ticks":9015,"time":{"ms":47},"value":9015},"user":{"ticks":7625,"time":{"ms":16}}},"handles":{"open":165},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":540501},"version":"7.17.3"},"memstats":{"gc_next":21075760,"memory_alloc":12255288,"memory_total":1273869664,"rss":54386688},"runtime":{"goroutines":49}},"filebeat":{"events":{"added":65,"done":65},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":65,"active":0,"batches":1,"total":65},"read":{"bytes":6},"write":{"bytes":3139}},"pipeline":{"clients":3,"events":{"active":0,"published":65,"total":65},"queue":{"acked":65}}},"registrar":{"states":{"current":3,"update":65},"writes":{"success":1,"total":1}}}}}
2022-09-13T08:43:45.438-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1421,"time":{"ms":31}},"total":{"ticks":9077,"time":{"ms":62},"value":9077},"user":{"ticks":7656,"time":{"ms":31}}},"handles":{"open":169},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":570508},"version":"7.17.3"},"memstats":{"gc_next":21075760,"memory_alloc":12574416,"memory_total":1274188792,"rss":54386688},"runtime":{"goroutines":49}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"active":0}},"pipeline":{"clients":3,"events":{"active":0}}},"registrar":{"states":{"current":3}}}}}
2022-09-13T08:44:15.447-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1421,"time":{"ms":16}},"total":{"ticks":9077,"time":{"ms":16},"value":9077},"user":{"ticks":7656}},"handles":{"open":169},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":600504},"version":"7.17.3"},"memstats":{"gc_next":21075760,"memory_alloc":14196312,"memory_total":1275810688,"rss":55074816},"runtime":{"goroutines":49}},"filebeat":{"events":{"added":41,"done":41},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":41,"active":0,"batches":1,"total":41},"read":{"bytes":6},"write":{"bytes":2210}},"pipeline":{"clients":3,"events":{"active":0,"published":41,"total":41},"queue":{"acked":41}}},"registrar":{"states":{"current":3,"update":41},"writes":{"success":1,"total":1}}}}}
2022-09-13T08:44:45.446-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1453,"time":{"ms":16}},"total":{"ticks":9140,"time":{"ms":47},"value":9140},"user":{"ticks":7687,"time":{"ms":31}}},"handles":{"open":167},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":630501},"version":"7.17.3"},"memstats":{"gc_next":21015184,"memory_alloc":10508752,"memory_total":1276136952,"rss":53592064},"runtime":{"goroutines":49}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"active":0}},"pipeline":{"clients":3,"events":{"active":0}}},"registrar":{"states":{"current":3}}}}}
2022-09-13T08:45:15.449-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1453},"total":{"ticks":9140,"value":9140},"user":{"ticks":7687}},"handles":{"open":169},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":660499},"version":"7.17.3"},"memstats":{"gc_next":21015184,"memory_alloc":12191328,"memory_total":1277819528,"rss":53927936},"runtime":{"goroutines":49}},"filebeat":{"events":{"added":56,"done":56},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":56,"active":0,"batches":1,"total":56},"read":{"bytes":6},"write":{"bytes":2774}},"pipeline":{"clients":3,"events":{"active":0,"published":56,"total":56},"queue":{"acked":56}}},"registrar":{"states":{"current":3,"update":56},"writes":{"success":1,"total":1}}}}}
2022-09-13T08:45:45.465-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1453},"total":{"ticks":9140,"value":9140},"user":{"ticks":7687}},"handles":{"open":169},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":690511},"version":"7.17.3"},"memstats":{"gc_next":21015184,"memory_alloc":12515664,"memory_total":1278143864,"rss":53927936},"runtime":{"goroutines":49}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"active":0}},"pipeline":{"clients":3,"events":{"active":0}}},"registrar":{"states":{"current":3}}}}}
2022-09-13T08:46:15.458-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1468,"time":{"ms":31}},"total":{"ticks":9155,"time":{"ms":31},"value":9155},"user":{"ticks":7687}},"handles":{"open":167},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":720504},"version":"7.17.3"},"memstats":{"gc_next":21015184,"memory_alloc":14216896,"memory_total":1279845096,"rss":54558720},"runtime":{"goroutines":49}},"filebeat":{"events":{"added":62,"done":62},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":62,"active":0,"batches":1,"total":62},"read":{"bytes":6},"write":{"bytes":2958}},"pipeline":{"clients":3,"events":{"active":0,"published":62,"total":62},"queue":{"acked":62}}},"registrar":{"states":{"current":3,"update":62},"writes":{"success":1,"total":1}}}}}
2022-09-13T08:46:45.457-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1484},"total":{"ticks":9171,"value":9171},"user":{"ticks":7687}},"handles":{"open":169},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":750498},"version":"7.17.3"},"memstats":{"gc_next":21143760,"memory_alloc":10568096,"memory_total":1280166128,"rss":53723136},"runtime":{"goroutines":49}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"active":0}},"pipeline":{"clients":3,"events":{"active":0}}},"registrar":{"states":{"current":3}}}}}
2022-09-13T08:47:15.456-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1500,"time":{"ms":16}},"total":{"ticks":9187,"time":{"ms":16},"value":9187},"user":{"ticks":7687}},"handles":{"open":169},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":780497},"version":"7.17.3"},"memstats":{"gc_next":21143760,"memory_alloc":12459080,"memory_total":1282057112,"rss":53727232},"runtime":{"goroutines":49}},"filebeat":{"events":{"added":111,"done":111},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":111,"active":0,"batches":1,"total":111},"read":{"bytes":6},"write":{"bytes":4787}},"pipeline":{"clients":3,"events":{"active":0,"published":111,"total":111},"queue":{"acked":111}}},"registrar":{"states":{"current":3,"update":111},"writes":{"success":1,"total":1}}}}}
2022-09-13T08:47:45.459-0400 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1500},"total":{"ticks":9187,"value":9187},"user":{"ticks":7687}},"handles":{"open":167},"info":{"ephemeral_id":"ef115b3f-bc86-4802-8e8d-cb95d43930ac","uptime":{"ms":810502},"version":"7.17.3"},"memstats":{"gc_next":21143760,"memory_alloc":12859752,"memory_total":1282457784,"rss":53731328},"runtime":{"goroutines":49}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"active":0}},"pipeline":{"clients":3,"events":{"active":0}}},"registrar":{"states":{"current":3}}}}}
```
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [September 13, 2022, 2:12pm UTC](https://discuss.elastic.co/t/after-filebeat-upgrades-preserving-registry-it-sending-duplicate-events/313701/6 "2022-09-13T14:12:47Z")
> [@krish0608](#):
>
> ```auto
> 2022-09-13T08:34:15.373-0400 INFO registrar/migrate.go:129 Migrate registry file to registry directory
> 2022-09-13T08:34:15.373-0400 INFO registrar/migrate.go:134 Move registry file to backup file: C:\\Program Files\\filebeat\\data\\registry.bak
> 2022-09-13T08:34:15.373-0400 INFO registrar/migrate.go:163 No registry home found. Create: C:\Program Files\filebeat\data\registry\filebeat
> 2022-09-13T08:34:15.374-0400 INFO registrar/migrate.go:171 Initialize registry meta file
> 2022-09-13T08:34:15.375-0400 INFO [monitoring] log/log.go:142 Starting metrics logging every 30s
> 2022-09-13T08:34:15.376-0400 INFO registrar/migrate.go:151 Migrate old registry file to new data file
> 2022-09-13T08:34:15.376-0400 INFO registrar/migrate.go:185 Migrate registry version 0 to version 1
> 2022-09-13T08:34:15.377-0400 INFO memlog/store.go:119 Loading data file of 'C:\Program Files\filebeat\data\registry\filebeat' succeeded. Active transaction id=0
>
> ```
These look odd to me..are those log lines present every time you start filebeat?
Particularly this
`No registry home found.`
When you upgraded did you use the latest filebeat.yml or did you leave the old 5.6 filebeat.yml?
@andrewkroh do you have any thoughts?
---
### Author:  [@krish0608](https://discuss.elastic.co/u/krish0608)
#### Post date: [September 13, 2022, 3:28pm UTC](https://discuss.elastic.co/t/after-filebeat-upgrades-preserving-registry-it-sending-duplicate-events/313701/7 "2022-09-13T15:28:03Z")
No after upgrade it was my latest filebeat.yml. I was only preserving old registry file and after upgrade it automatically convert registry file with regsitry folder and under that it add older offset values.
And regarding that error "No registry home found" I have resolved that i was doing some testing because of that we got that error.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [September 13, 2022, 3:49pm UTC](https://discuss.elastic.co/t/after-filebeat-upgrades-preserving-registry-it-sending-duplicate-events/313701/8 "2022-09-13T15:49:33Z")
> [@krish0608](#):
>
> Are you receiving duplicate events ongoing?- The process i followed was i checked no of Logs count in Kibana before upgrade and after upgrade it resend current date events from Logfile. And yes it the first time after the upgrade.
Thanks...
So to be clear _ **every** _ time you restart filebeat on windows it is reloading **ALL** the log files from the beginning..
Or does it only re-load the ALL log files at **Once** at startup and then ongoing is fine.
I am still unclear... apologies.
---
### Author:  [@krish0608](https://discuss.elastic.co/u/krish0608)
#### Post date: [September 14, 2022, 6:59am UTC](https://discuss.elastic.co/t/after-filebeat-upgrades-preserving-registry-it-sending-duplicate-events/313701/9 "2022-09-14T06:59:42Z")
Yes it only re-load the ALL log files at **Once** at startup and then ongoing is fine.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [September 14, 2022, 2:05pm UTC](https://discuss.elastic.co/t/after-filebeat-upgrades-preserving-registry-it-sending-duplicate-events/313701/10 "2022-09-14T14:05:33Z")
It looks to me that your original 5.6.5 registry file / data is missing or perhaps corrupted and thus a new registry is created.
Do you have a copy of the old registry data / file?
Unfortunately I am not an expert on the internals of registry files especially ones that are from 5.6 which is years old... but it you can find it perhaps you can copy it into the Directory before you start filebeat.
`C:\Program Files\filebeat\data\registry\filebeat`
---
### Author:  [@krish0608](https://discuss.elastic.co/u/krish0608)
#### Post date: [September 14, 2022, 2:34pm UTC](https://discuss.elastic.co/t/after-filebeat-upgrades-preserving-registry-it-sending-duplicate-events/313701/11 "2022-09-14T14:34:02Z")
Yes we are following same steps. we copy registry file of 5.6.5 to new location. And after upgrade before restarting filebeat service we copy 5.6.5 registry file to / data directory.
And after restart it automatically convert 5.6.5 registry file into registry folder( adding offset of5.6.5).
I am guessing there might some configuration i need to add to prevent it from to read file again from strach. After upgrade it should only look for delta or newly added Logs from file.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [September 14, 2022, 2:47pm UTC](https://discuss.elastic.co/t/after-filebeat-upgrades-preserving-registry-it-sending-duplicate-events/313701/12 "2022-09-14T14:47:48Z")
> [@krish0608](#):
>
> And after restart it automatically convert 5.6.5 registry file into registry folder( adding offset of5.6.5).
To me it seems like it is not reading the old registry correctly ... (obvious 🙂 )
Not sure if you are going to get it to work correctly because of the very old version.
Perhaps you could look at this setting to help with the problem
> **[filestream input | Filebeat Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#filebeat-input-filestream-ignore-older)**
> ##### `ignore_older`
>
> If this option is enabled, Filebeat ignores any files that were modified before the specified timespan. Configuring `ignore_older` can be especially useful if you keep log files for a long time. For example, if you want to start Filebeat, but only want to send the newest files and files from last week, you can configure this option.
---
### Author:  [@krish0608](https://discuss.elastic.co/u/krish0608)
#### Post date: [September 14, 2022, 3:23pm UTC](https://discuss.elastic.co/t/after-filebeat-upgrades-preserving-registry-it-sending-duplicate-events/313701/13 "2022-09-14T15:23:32Z")
We are already using ignore\_older:24h but its not helped us from data duplicacy.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [September 14, 2022, 4:19pm UTC](https://discuss.elastic.co/t/after-filebeat-upgrades-preserving-registry-it-sending-duplicate-events/313701/14 "2022-09-14T16:19:00Z")
Apologies, I am not sure what to tell you at this point... if you are getting duplicated events in the last 24 hours.
It could be a bug in the conversion from 5.6 to 7.17 on windows... 5.6 is several years old / very very old.
You could try running a 6.x filebeat then a 7.x filebeat and see if the registry conversion works.
I do not think there is a simple answer at this point, you may just need to de-dup your data another way.
Perhaps looking at one of the older threads will help
> [@Duplicate events with Filebeat on windows on service restart](https://discuss.elastic.co/t/duplicate-events-with-filebeat-on-windows-on-service-restart/78743/12):
>
> Ok, so in 5.2 we can "solve" the problem by telling the user to use option 2 or 3 for the path config and in 5.3 it should be solved already.
---
### Author:  [@krish0608](https://discuss.elastic.co/u/krish0608)
#### Post date: [September 16, 2022, 7:12am UTC](https://discuss.elastic.co/t/after-filebeat-upgrades-preserving-registry-it-sending-duplicate-events/313701/15 "2022-09-16T07:12:00Z")
Thanks @stephenb for your support. Do you have any idea how to tell filebeat/what i have to define in my configuration so that it should read only newly logs instead of whole file i am already using ignore\_older:24h.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [September 16, 2022, 4:15pm UTC](https://discuss.elastic.co/t/after-filebeat-upgrades-preserving-registry-it-sending-duplicate-events/313701/16 "2022-09-16T16:15:01Z")
> [@krish0608](#):
>
> ` ignore_older: 25h`
Perhaps you are not putting the `ignore_older` in the right place in your ymls
I can not tell with the .yml snippets above where you actually have `ignore_older` set those snippets are not consistent / malformed... the indentation is wrong.... etc...
Also you have just this.. with nothing under it... I can not tell what you are trying to do
```auto
filebeat.inputs:
- type: filestream
```
Then the second snippet of code is malformed and I do not know what it is
What I can tell you .... I tested and `ignore_older` and it worked as designed
This is a test I just ran and the file is older than 24h and it did not read it
I took out `ignore_older` and it read it... so it does work. (I do not have a windows box so I can not test there but I think there would be a bug or something filed if it did not work, I can not find one with a quick look)
```auto
filebeat.inputs:
- type: filestream
id: my-filestream-id
enabled: true
ignore_older: 24h
paths:
- /Users/sbrown/workspace/sample-data/discuss/container-mixed/*.log
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 14, 2022, 6:15pm UTC](https://discuss.elastic.co/t/after-filebeat-upgrades-preserving-registry-it-sending-duplicate-events/313701/17 "2022-10-14T18:15:20Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.