Cant publish but harverst with 27w files

when start filebeat (1.2.3) whith the path which have 270000 files but the size only 3G,it just harvester no publish.
the dug log:

below is the config:

          - /data1/logs/**/*.txt
     input_type: log
     close_older: 1m
     document_type: xzb_logs
     scan_frequency: 30s
     backoff: 1h
     max_backoff: 24h
     spool_size: 16048
    idle_timeout: 5s
     registry_file: /usr/local/filebeat/filebeat-1.2.3-x86_64/filebeat_registry
           hosts: [""]
      worker: 2
      to_files: true
      path: /usr/local/filebeat/logs
       name: mybeat.log
       rotateeverybytes: 10485760
      keepfiles: 10
       level: debug

when i change the path with less files ,it works well.
my logs files uploaded by other system and do not update again

oh no,it start publishing after about 1 h. but i config the idle_timeout with 5s, it should pulish per 5s, i cant understant it .

  • Is there a reason you modified to the backoff variables?
  • Could you try the nightly build to see if the problem persists there:
  • What is the maximum number of open files you allow? With how many files does it still work?
  • Can you post the log message as text instead of a screenshot? This makes it easier readable and searchable.

This topic was automatically closed after 21 days. New replies are no longer allowed.