Cant publish but harverst with 27w files

Elastic Stack Beats
1

when start filebeat （1.2.3） whith the path which have 270000 files but the size only 3G,it just harvester no publish.
the dug log:

Pasted image1107×881 46.4 KB

below is the config:

filebeat:
paths:
     prospectors:
          - /data1/logs/**/*.txt
     input_type: log
     close_older: 1m
     document_type: xzb_logs
     scan_frequency: 30s
     backoff: 1h
     max_backoff: 24h
     spool_size: 16048
    idle_timeout: 5s
     registry_file: /usr/local/filebeat/filebeat-1.2.3-x86_64/filebeat_registry
output:
      logstash:
           hosts: ["10.162.10.34:5044"]
      worker: 2
logging:
      to_files: true
      path: /usr/local/filebeat/logs
       name: mybeat.log
       rotateeverybytes: 10485760
      keepfiles: 10
       level: debug

when i change the path with less files ,it works well.
my logs files uploaded by other system and do not update again

==============================
oh no,it start publishing after about 1 h. but i config the idle_timeout with 5s, it should pulish per 5s, i cant understant it .

2
  • Is there a reason you modified to the backoff variables?
  • Could you try the nightly build to see if the problem persists there: https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/
  • What is the maximum number of open files you allow? With how many files does it still work?
  • Can you post the log message as text instead of a screenshot? This makes it easier readable and searchable.
3

This topic was automatically closed after 21 days. New replies are no longer allowed.