# Very long to load prospectors and start harvesting

**URL:** <https://discuss.elastic.co/t/very-long-to-load-prospectors-and-start-harvesting/86625>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 22, 2017, 8:27am UTC](https://discuss.elastic.co/t/very-long-to-load-prospectors-and-start-harvesting/86625 "2017-05-22T08:27:04Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![ogauchard](https://avatars.discourse-cdn.com/v4/letter/o/3da27b/32.png) [@ogauchard](https://discuss.elastic.co/u/ogauchard)\
**Post date:** [May 22, 2017, 8:27am UTC](https://discuss.elastic.co/t/very-long-to-load-prospectors-and-start-harvesting/86625/1 "2017-05-22T08:27:04Z")

</div>

Hi all,

We are encountering a strange behaviour with filebeat.

Let me explain our use case as it is not the typical goal of the filebeat log prospector :

- We use filebeat with 5 log prospectors.

- Each prospector observes one or more directories using \* wildcard in directory path and filename : C:\root\sub\__\_.csv for instance

- Each directory contains from 100 to 43.000 files with a total of 125.000 files

- 2 prospectors are harvesting CSV files containing only one row

- The 3 other prospectors are reading XML or custom format files using multiline. For these files, **one file = one event**

- The CSV files are by far the most numerous

- Each prospector has its `ignore_older` property set to `24h`

- We are on a Windows Server 2012 R2 server

The problem is when we have to change filebeat config and restart the service.

Last time, I started filebeat at **9 AM** and in the log file I can see that it finished loading prospectors and starting harvesting at **11 PM**!

During all the "loading" time, I only see lines like these one :

```
2017-05-17T18:24:50+02:00 INFO No non-zero metrics in the last 30s
2017-05-17T18:25:20+02:00 INFO Non-zero metrics in the last 30s: publish.events=3 registrar.states.update=3 registrar.writes=3
2017-05-17T18:25:50+02:00 INFO Non-zero metrics in the last 30s: registrar.states.update=4 registrar.writes=4 publish.events=4

```

The problem seems to be related to filebeat registry data loading. If I delete the registry data and let filebeat consume all data (only the last 24h due to configured `ignore_older`) the harvesting starts very quickly.

Can you give me some explanations about this behaviour and if there is a way to reduce the loading time ?

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [May 23, 2017, 12:37pm UTC](https://discuss.elastic.co/t/very-long-to-load-prospectors-and-start-harvesting/86625/2 "2017-05-23T12:37:29Z")

</div>

Which filebeat version are you using?  
Can you share your full config?  
Are you using any clean\_\* options?

---

<div class="post-metadata">

**Author:** ![ogauchard](https://avatars.discourse-cdn.com/v4/letter/o/3da27b/32.png) [@ogauchard](https://discuss.elastic.co/u/ogauchard)\
**Post date:** [May 23, 2017, 12:48pm UTC](https://discuss.elastic.co/t/very-long-to-load-prospectors-and-start-harvesting/86625/3 "2017-05-23T12:48:23Z")

</div>

Thank you Nicolas for answering.

We are using filebeat 5.0.2

Here is our config file (I have changed the path and document\_type for confidentiality reasons):

```
filebeat:
  prospectors:
    -
      paths:
        - D:\_data\*_TO_FOO\*.csv
        - D:\_data\FOO_TO_BAR\ARCHIVE\*.CSV
        - D:\_data\FOO2_TO_BAR\EXPORT\*.csv
      ignore_older: 24h
      document_type: document_type_A
      scan_frequency: 5s

    -
      paths:
        - D:\_data\FOO3_TO_BAR\EXPORT\*.xml
      ignore_older: 24h
      fields_under_root: true
      fields:
        SourceSystem: FOO3
      document_type: document_type_A
      exclude_lines: ['^<\?xml', '^<Document']
      multiline:
        pattern: '^[[:space:]]*<Node'
        negate: true
        match: after
        max_lines: 5000

    -
      paths:
        - D:\_data\FOO_TO_*\*.csv
      ignore_older: 24h
      document_type: document_type_B
      scan_frequency: 5s

    -
      paths:
        - D:\_data\AC\Processed\*.qid
      ignore_older: 24h
      document_type: document_type_C
      scan_frequency: 5s
      multiline:
        pattern: ^EXENAME
        negate: true
        match: after

    -
      paths:
        - D:\_data\AT\Processed\*.qid
      ignore_older: 24h
      document_type: document_type_D
      scan_frequency: 5s
      multiline:
        pattern: ^EXENAME
        negate: true
        match: after

output:
  logstash:
    hosts: ["logstash-int:5044"]

logging:
  files:
    path: D:\filebeat\5.0.2\log
    rotateeverybytes: 10485760 # = 10MB

```

You can see that we are not using any clean\_\* or close\_\* option. And I think this is the problem...

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [May 23, 2017, 2:04pm UTC](https://discuss.elastic.co/t/very-long-to-load-prospectors-and-start-harvesting/86625/4 "2017-05-23T14:04:29Z")

</div>

What is the size of your registry file? Do you have spinning disks or reasonable fast SSD's? How many entries do you have in the registry file? I assume this could become an issue at some point.

---

<div class="post-metadata">

**Author:** ![ogauchard](https://avatars.discourse-cdn.com/v4/letter/o/3da27b/32.png) [@ogauchard](https://discuss.elastic.co/u/ogauchard)\
**Post date:** [May 23, 2017, 2:12pm UTC](https://discuss.elastic.co/t/very-long-to-load-prospectors-and-start-harvesting/86625/5 "2017-05-23T14:12:34Z")

</div>

The registry file size is 2.6 MB

There is almost 11K entries in the registry file.

Don't really know about disks. It is a virtual server with shared storage. (vmware)

---

<div class="post-metadata">

**Author:** ![ogauchard](https://avatars.discourse-cdn.com/v4/letter/o/3da27b/32.png) [@ogauchard](https://discuss.elastic.co/u/ogauchard)\
**Post date:** [May 24, 2017, 7:45am UTC](https://discuss.elastic.co/t/very-long-to-load-prospectors-and-start-harvesting/86625/6 "2017-05-24T07:45:24Z")

</div>

Well, I ommit a quite important detail. Filebeat is on a dedicated server and access files through network drives.

Can it be the cause of our issue ?

---

<div class="post-metadata">

**Author:** ![ogauchard](https://avatars.discourse-cdn.com/v4/letter/o/3da27b/32.png) [@ogauchard](https://discuss.elastic.co/u/ogauchard)\
**Post date:** [May 24, 2017, 9:46am UTC](https://discuss.elastic.co/t/very-long-to-load-prospectors-and-start-harvesting/86625/7 "2017-05-24T09:46:59Z")

</div>

@ruflin This topic (86625) is related to another topic I have created (86782). Could you please delete both topics because I want to create a new one with more details and elements based on my recent tests.  
Thank you

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [May 26, 2017, 6:05am UTC](https://discuss.elastic.co/t/very-long-to-load-prospectors-and-start-harvesting/86625/8 "2017-05-26T06:05:10Z")

</div>

Just comment on the existing topics with a link to the new topic.

---

<div class="post-metadata">

**Author:** ![ogauchard](https://avatars.discourse-cdn.com/v4/letter/o/3da27b/32.png) [@ogauchard](https://discuss.elastic.co/u/ogauchard)\
**Post date:** [May 29, 2017, 8:07am UTC](https://discuss.elastic.co/t/very-long-to-load-prospectors-and-start-harvesting/86625/9 "2017-05-29T08:07:04Z")

</div>

This topic is considered as closed.

It is merged with this topic : [https://discuss.elastic.co/t/recommended-configuration-for-write-once-files/86782/7](https://discuss.elastic.co/t/recommended-configuration-for-write-once-files/86782/7)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 26, 2017, 8:07am UTC](https://discuss.elastic.co/t/very-long-to-load-prospectors-and-start-harvesting/86625/10 "2017-06-26T08:07:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.