Filebeat loss log data

Filebeat not collect file which has being marked inactived. But the file is always update intime.

We use filebeat to collect application's log to logstash.

config:

    filebeat.inputs:
      - type: log
        enabled: true
        paths:
          - /home/xxxlogs/logs/sys/*/*.log
          - /home/xxxlogs/logs/sys/*.log
        max_bytes: 1048576
        ignore_older: 168h
        multiline.pattern: '^20\d{2}'
        multiline.negate: true
        multiline.match: after
        multiline.max_lines: 500
        multiline.timeout: 5
    filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false
      reload.period: 10s
    queue.mem:
      events: 200
      flush.min_events: 100
    output.logstash:
      hosts: ['x.x.x.x:5048']
      bulk_max_size: 100
    http:
      enabled: true
      host: localhost
      port: 5066

    # filebeat logs setting
    logging.level: info
    logging.to_files: true
    logging.files:
      path: /home/xxxlogs/applogs/filebeat
      name: filebeat.log
      keepfiles: 7
      permissions: 0644
      rotateeverybytesedit: 209715200

in filebeat-log:
We found the log was be marked inactived after close_inactive time .

2020-05-12T00:00:09.101+0800    INFO    log/harvester.go:251    Harvester started for file: /home/xxxlogs/logs/sys/xxx/http-server.log"

2020-05-12T00:05:05.664+0800    INFO    log/harvester.go:276    File is inactive: /home/xxxlogs/logs/sys/xxx/http-server.log. Closing because close_inactive of 5m0s reached.

but actual file are always being updated...

After we restart the application.
The log in file unexpectedly can be collected. but the filebeat-log has no "Harvester started for file: " record.
No rotate log event occured!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.