Speed up filebeat registrar

Elastic Stack Beats
1

I'm searching through 200000 files on a server (24 logical cores, 48GB RAM) and want to collect logs only for the previous 24 hours:

My filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - '/data/logs/A[0-9]*/*/scheduler/log.txt'
    - '/data/logs/A[0-9]*/*/tables/*.txt'
    - '/data/logs/A[0-9]*/*/server.txt'
    - '/data/logs/A[0-9]*/*/errors_fatal.txt'
    - '/data/logs/A[0-9]*/*/communicator/log.txt'
  encoding: utf-8
  multiline.pattern: '^\d{8} \d{2}:\d{2}:\d{2}\.\d{3}'
  multiline.negate: true
  multiline.match: after
  close_renamed: true
  harvester_limit: 1024
  ignore_older: 24h

output.logstash:
  hosts: ["log.ff.local:5044"]

The main problem is if I restart filebeat it doesn't detect new files as it busy with an old ones. I see the following in a debug log:

2018-12-14T12:56:49.542+0700    DEBUG    [registrar]    registrar/registrar.go:393    Registry file updated. 182331 states written.
2018-12-14T12:56:49.542+0700    DEBUG    [registrar]    registrar/registrar.go:345    Processing 1 events
2018-12-14T12:56:49.542+0700    DEBUG    [registrar]    registrar/registrar.go:315    Registrar state updates processed. Count: 1
2018-12-14T12:56:49.542+0700    DEBUG    [acker]    beater/acker.go:64    stateful ack    {"count": 1}
2018-12-14T12:56:49.543+0700    DEBUG    [input]    file/states.go:68    New state added for /data/logs/A19/21001166 20181105 1604/tables/11_2949370.txt
2018-12-14T12:56:49.546+0700    DEBUG    [registrar]    registrar/registrar.go:335    Registrar states cleaned up. Before: 182331, After: 182331, Pending: 0
2018-12-14T12:56:49.572+0700    DEBUG    [registrar]    registrar/registrar.go:400    Write registry file: /var/lib/filebeat/registry
2018-12-14T12:56:50.257+0700    DEBUG    [registrar]    registrar/registrar.go:393    Registry file updated. 182331 states written.
2018-12-14T12:56:50.257+0700    DEBUG    [registrar]    registrar/registrar.go:345    Processing 1 events
2018-12-14T12:56:50.257+0700    DEBUG    [registrar]    registrar/registrar.go:315    Registrar state updates processed. Count: 1
2018-12-14T12:56:50.257+0700    DEBUG    [acker]    beater/acker.go:64    stateful ack    {"count": 1}
2018-12-14T12:56:50.257+0700    DEBUG    [input]    file/states.go:68    New state added for /data/logs/A31/21001432 20181207 1827/tables/41_8323722.txt
2018-12-14T12:56:50.261+0700    DEBUG    [registrar]    registrar/registrar.go:335    Registrar states cleaned up. Before: 182331, After: 182331, Pending: 0
2018-12-14T12:56:50.280+0700    DEBUG    [registrar]    registrar/registrar.go:400    Write registry file: /var/lib/filebeat/registry
2018-12-14T12:56:51.008+0700    DEBUG    [registrar]    registrar/registrar.go:393    Registry file updated. 182331 states written.
2018-12-14T12:56:51.009+0700    DEBUG    [registrar]    registrar/registrar.go:345    Processing 1 events
2018-12-14T12:56:51.009+0700    DEBUG    [registrar]    registrar/registrar.go:315    Registrar state updates processed. Count: 1
2018-12-14T12:56:51.009+0700    DEBUG    [acker]    beater/acker.go:64    stateful ack    {"count": 1}
2018-12-14T12:56:51.009+0700    DEBUG    [input]    file/states.go:68    New state added for /data/logs/A1019/21002965 20181117 0258/tables/4985_2556556.txt
2018-12-14T12:56:51.011+0700    DEBUG    [registrar]    registrar/registrar.go:335    Registrar states cleaned up. Before: 182331, After: 182331, Pending: 0
2018-12-14T12:56:51.029+0700    DEBUG    [registrar]    registrar/registrar.go:400    Write registry file: /var/lib/filebeat/registry
2018-12-14T12:56:51.725+0700    DEBUG    [registrar]    registrar/registrar.go:393    Registry file updated. 182331 states written.

So, according to the debug log, the maximum speed for registrar is 2 files per second and approximately it will take 1-2 days to check all files which is so long (even in comparison with deleting the registry file and restart filebeat). Is it possible to speed up filebeat registrar?

Also, is it okay that the 182331 states written string is not changed and I see the same number for different files?

2

Hello @Andrey, we are currently in the process of refactoring the registrar to make it faster and more resilient. Some of the fixes should already be in Filebeat, what version are you using?

3

Hi, I'm using the latest (6.5.3) version. Meanwhile, the issue was solved after I added filebeat.registry_flush: 30s. For some reason I missed that part in the documentation.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.