Very long to load prospectors and start harvesting

Elastic Stack Beats
1

Hi all,

We are encountering a strange behaviour with filebeat.

Let me explain our use case as it is not the typical goal of the filebeat log prospector :

  • We use filebeat with 5 log prospectors.

  • Each prospector observes one or more directories using * wildcard in directory path and filename : C:\root\sub_\.csv for instance

  • Each directory contains from 100 to 43.000 files with a total of 125.000 files

  • 2 prospectors are harvesting CSV files containing only one row

  • The 3 other prospectors are reading XML or custom format files using multiline. For these files, one file = one event

  • The CSV files are by far the most numerous

  • Each prospector has its ignore_older property set to 24h

  • We are on a Windows Server 2012 R2 server

The problem is when we have to change filebeat config and restart the service.

Last time, I started filebeat at 9 AM and in the log file I can see that it finished loading prospectors and starting harvesting at 11 PM !

During all the "loading" time, I only see lines like these one :

2017-05-17T18:24:50+02:00 INFO No non-zero metrics in the last 30s
2017-05-17T18:25:20+02:00 INFO Non-zero metrics in the last 30s: publish.events=3 registrar.states.update=3 registrar.writes=3
2017-05-17T18:25:50+02:00 INFO Non-zero metrics in the last 30s: registrar.states.update=4 registrar.writes=4 publish.events=4

The problem seems to be related to filebeat registry data loading. If I delete the registry data and let filebeat consume all data (only the last 24h due to configured ignore_older) the harvesting starts very quickly.

Can you give me some explanations about this behaviour and if there is a way to reduce the loading time ?

2

Which filebeat version are you using?
Can you share your full config?
Are you using any clean_* options?

3

Thank you Nicolas for answering.

We are using filebeat 5.0.2

Here is our config file (I have changed the path and document_type for confidentiality reasons):

filebeat:
  prospectors:
    -
      paths:
        - D:\_data\*_TO_FOO\*.csv
        - D:\_data\FOO_TO_BAR\ARCHIVE\*.CSV
        - D:\_data\FOO2_TO_BAR\EXPORT\*.csv
      ignore_older: 24h
      document_type: document_type_A
      scan_frequency: 5s

    -
      paths:
        - D:\_data\FOO3_TO_BAR\EXPORT\*.xml
      ignore_older: 24h
      fields_under_root: true
      fields:
        SourceSystem: FOO3
      document_type: document_type_A
      exclude_lines: [ '^<\?xml', '^<Document' ]
      multiline:
        pattern: '^[[:space:]]*<Node'
        negate: true
        match: after
        max_lines: 5000

    -
      paths:
        - D:\_data\FOO_TO_*\*.csv
      ignore_older: 24h
      document_type: document_type_B
      scan_frequency: 5s

    -
      paths:
        - D:\_data\AC\Processed\*.qid
      ignore_older: 24h
      document_type: document_type_C
      scan_frequency: 5s
      multiline:
        pattern: ^EXENAME
        negate: true
        match: after

    -
      paths:
        - D:\_data\AT\Processed\*.qid
      ignore_older: 24h
      document_type: document_type_D
      scan_frequency: 5s
      multiline:
        pattern: ^EXENAME
        negate: true
        match: after

output:
  logstash:
    hosts: ["logstash-int:5044"]

logging:
  files:
    path: D:\filebeat\5.0.2\log
    rotateeverybytes: 10485760 # = 10MB

You can see that we are not using any clean_* or close_* option. And I think this is the problem...

4

What is the size of your registry file? Do you have spinning disks or reasonable fast SSD's? How many entries do you have in the registry file? I assume this could become an issue at some point.

5

The registry file size is 2.6 MB

There is almost 11K entries in the registry file.

Don't really know about disks. It is a virtual server with shared storage. (vmware)

6

Well, I ommit a quite important detail. Filebeat is on a dedicated server and access files through network drives.

Can it be the cause of our issue ?

7

@ruflin This topic (86625) is related to another topic I have created (86782). Could you please delete both topics because I want to create a new one with more details and elements based on my recent tests.
Thank you

8

Just comment on the existing topics with a link to the new topic.

9

This topic is considered as closed.

It is merged with this topic : https://discuss.elastic.co/t/recommended-configuration-for-write-once-files/86782/7

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.