Filebeat-6.2.4 no send finish in time


(Charliecen) #1

Problem description:

More than 2000 files are generated per hour in the multi-level directory, I only need to log in an hour, so I'm configuration ignore_older: 1 h, but after an hour, which is 1 hour before the file is not sent. How can I solve this problem?

config file

filebeat.prospectors:
- input_type: log
  paths:
    - /vol2/ZT2M/*/*/*.log.*
  scan_frequency: 10s
  ignore_older: 1h
  close_eof: true
  close_timeout: 3h
  close_interval: 5m
  close_removed: true
  clean_removed: true
  clean_inactive: 3h
  exclude_lines: ["存储至", "压缩档案数据成功"]
  include_lines: ['^\d{6}-\d{2}:\d{2}:\d{2}']
output.logstash:
  hosts: ["127.0.0.1:5044"]

Pictures show

qq20180510-170655

The logs are getting less and less from 13:00 to 13:59.


(Christian Dahlqvist) #2

What does your Logstash configuration look like? What is the specification of your Elasticsearch cluster?


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.