# Cant publish but harverst with 27w files

**URL:** https://discuss.elastic.co/t/cant-publish-but-harverst-with-27w-files/55007
**Category:** Beats
**Tags:** filebeat
**Created:** [July 8, 2016, 2:14am UTC](https://discuss.elastic.co/t/cant-publish-but-harverst-with-27w-files/55007 "2016-07-08T02:14:05Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![xiongzhichaoelk](https://avatars.discourse-cdn.com/v4/letter/x/3d9bf3/32.png) [@xiongzhichaoelk](https://discuss.elastic.co/u/xiongzhichaoelk)
#### Post date: [July 8, 2016, 2:14am UTC](https://discuss.elastic.co/t/cant-publish-but-harverst-with-27w-files/55007/1 "2016-07-08T02:14:05Z")

</div>

when start filebeat （1.2.3） whith the path which have 270000 files but the size only 3G,it just harvester no publish.  
the dug log:

 ![](https://us1.discourse-cdn.com/elastic/original/2X/c/cea67f53a9e9946bb21c0a62841df96ea8ef98c9.png)  
below is the config:

```auto
filebeat:
paths:
     prospectors:
          - /data1/logs/**/*.txt
     input_type: log
     close_older: 1m
     document_type: xzb_logs
     scan_frequency: 30s
     backoff: 1h
     max_backoff: 24h
     spool_size: 16048
    idle_timeout: 5s
     registry_file: /usr/local/filebeat/filebeat-1.2.3-x86_64/filebeat_registry
output:
      logstash:
           hosts: ["10.162.10.34:5044"]
      worker: 2
logging:
      to_files: true
      path: /usr/local/filebeat/logs
       name: mybeat.log
       rotateeverybytes: 10485760
      keepfiles: 10
       level: debug

```

when i change the path with less files ,it works well.  
my logs files uploaded by other system and do not update again

==============================  
oh no,it start publishing after about 1 h. but i config the idle\_timeout with 5s, it should pulish per 5s, i cant understant it .

---

<div class="post-metadata">

### Author: ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [July 11, 2016, 9:51am UTC](https://discuss.elastic.co/t/cant-publish-but-harverst-with-27w-files/55007/2 "2016-07-11T09:51:44Z")

</div>

- Is there a reason you modified to the backoff variables?
- Could you try the nightly build to see if the problem persists there: [https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/](https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/)
- What is the maximum number of open files you allow? With how many files does it still work?
- Can you post the log message as text instead of a screenshot? This makes it easier readable and searchable.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 29, 2016, 2:14am UTC](https://discuss.elastic.co/t/cant-publish-but-harverst-with-27w-files/55007/3 "2016-07-29T02:14:05Z")

</div>

This topic was automatically closed after 21 days. New replies are no longer allowed.