# Speed up filebeat registrar
**URL:** https://discuss.elastic.co/t/speed-up-filebeat-registrar/160873
**Category:** Beats
**Tags:** filebeat
**Created:** [December 14, 2018, 10:00am UTC](https://discuss.elastic.co/t/speed-up-filebeat-registrar/160873 "2018-12-14T10:00:23Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@s1p](https://discuss.elastic.co/u/s1p)
#### Post date: [December 14, 2018, 10:00am UTC](https://discuss.elastic.co/t/speed-up-filebeat-registrar/160873/1 "2018-12-14T10:00:23Z")
I'm searching through 200000 files on a server (24 logical cores, 48GB RAM) and want to collect logs only for the previous 24 hours:
> **My filebeat.yml**
>
> ```
> filebeat.inputs:
> - type: log
> enabled: true
> paths:
> - '/data/logs/A[0-9]*/*/scheduler/log.txt'
> - '/data/logs/A[0-9]*/*/tables/*.txt'
> - '/data/logs/A[0-9]*/*/server.txt'
> - '/data/logs/A[0-9]*/*/errors_fatal.txt'
> - '/data/logs/A[0-9]*/*/communicator/log.txt'
> encoding: utf-8
> multiline.pattern: '^\d{8} \d{2}:\d{2}:\d{2}\.\d{3}'
> multiline.negate: true
> multiline.match: after
> close_renamed: true
> harvester_limit: 1024
> ignore_older: 24h
>
> output.logstash:
> hosts: ["log.ff.local:5044"]
>
> ```
The main problem is if I **restart** filebeat it doesn't detect new files as it busy with an old ones. I see the following in a debug log:
```
2018-12-14T12:56:49.542+0700 DEBUG [registrar] registrar/registrar.go:393 Registry file updated. 182331 states written.
2018-12-14T12:56:49.542+0700 DEBUG [registrar] registrar/registrar.go:345 Processing 1 events
2018-12-14T12:56:49.542+0700 DEBUG [registrar] registrar/registrar.go:315 Registrar state updates processed. Count: 1
2018-12-14T12:56:49.542+0700 DEBUG [acker] beater/acker.go:64 stateful ack {"count": 1}
2018-12-14T12:56:49.543+0700 DEBUG [input] file/states.go:68 New state added for /data/logs/A19/21001166 20181105 1604/tables/11_2949370.txt
2018-12-14T12:56:49.546+0700 DEBUG [registrar] registrar/registrar.go:335 Registrar states cleaned up. Before: 182331, After: 182331, Pending: 0
2018-12-14T12:56:49.572+0700 DEBUG [registrar] registrar/registrar.go:400 Write registry file: /var/lib/filebeat/registry
2018-12-14T12:56:50.257+0700 DEBUG [registrar] registrar/registrar.go:393 Registry file updated. 182331 states written.
2018-12-14T12:56:50.257+0700 DEBUG [registrar] registrar/registrar.go:345 Processing 1 events
2018-12-14T12:56:50.257+0700 DEBUG [registrar] registrar/registrar.go:315 Registrar state updates processed. Count: 1
2018-12-14T12:56:50.257+0700 DEBUG [acker] beater/acker.go:64 stateful ack {"count": 1}
2018-12-14T12:56:50.257+0700 DEBUG [input] file/states.go:68 New state added for /data/logs/A31/21001432 20181207 1827/tables/41_8323722.txt
2018-12-14T12:56:50.261+0700 DEBUG [registrar] registrar/registrar.go:335 Registrar states cleaned up. Before: 182331, After: 182331, Pending: 0
2018-12-14T12:56:50.280+0700 DEBUG [registrar] registrar/registrar.go:400 Write registry file: /var/lib/filebeat/registry
2018-12-14T12:56:51.008+0700 DEBUG [registrar] registrar/registrar.go:393 Registry file updated. 182331 states written.
2018-12-14T12:56:51.009+0700 DEBUG [registrar] registrar/registrar.go:345 Processing 1 events
2018-12-14T12:56:51.009+0700 DEBUG [registrar] registrar/registrar.go:315 Registrar state updates processed. Count: 1
2018-12-14T12:56:51.009+0700 DEBUG [acker] beater/acker.go:64 stateful ack {"count": 1}
2018-12-14T12:56:51.009+0700 DEBUG [input] file/states.go:68 New state added for /data/logs/A1019/21002965 20181117 0258/tables/4985_2556556.txt
2018-12-14T12:56:51.011+0700 DEBUG [registrar] registrar/registrar.go:335 Registrar states cleaned up. Before: 182331, After: 182331, Pending: 0
2018-12-14T12:56:51.029+0700 DEBUG [registrar] registrar/registrar.go:400 Write registry file: /var/lib/filebeat/registry
2018-12-14T12:56:51.725+0700 DEBUG [registrar] registrar/registrar.go:393 Registry file updated. 182331 states written.
```
So, according to the debug log, the maximum speed for registrar is 2 files per second and approximately it will take 1-2 days to check all files which is so long (even in comparison with deleting the registry file and restart filebeat). Is it possible to speed up filebeat registrar?
Also, is it okay that the `182331 states written` string is not changed and I see the same number for different files?
---
### Author:  [@pierhugues](https://discuss.elastic.co/u/pierhugues)
#### Post date: [December 17, 2018, 3:52pm UTC](https://discuss.elastic.co/t/speed-up-filebeat-registrar/160873/2 "2018-12-17T15:52:46Z")
Hello @Andrey, we are currently in the process of refactoring the registrar to make it faster and more resilient. Some of the fixes should already be in Filebeat, what version are you using?
---
### Author:  [@s1p](https://discuss.elastic.co/u/s1p)
#### Post date: [December 19, 2018, 8:39am UTC](https://discuss.elastic.co/t/speed-up-filebeat-registrar/160873/3 "2018-12-19T08:39:55Z")
Hi, I'm using the latest (6.5.3) version. Meanwhile, the issue was solved after I added `filebeat.registry_flush: 30s`. For some reason I missed that part in the [documentation](https://www.elastic.co/guide/en/beats/filebeat/6.5/configuration-general-options.html#_literal_registry_flush_literal).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 16, 2019, 8:54am UTC](https://discuss.elastic.co/t/speed-up-filebeat-registrar/160873/4 "2019-01-16T08:54:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.