Filebeats Deleted files and potentially missing data

We are using filebeats to ingest log files that are rotated hourly. This is sent to a load-balanced between 4 hosts running logstash behind HA proxy. I have been tweaking with the close_* options as I am seeing a huge build up of "deleted" files that are held by filebeats, eventhough the log files are cycled. As a result, I am seeing the follow pattern in our data, where on top of the hour, there's an influx of data, the events will start to drop off until the next hour. If I disable filebeats and the logging for a period of time, the data will flow solid for a while then the "drops" and pattern starts again. Any insight on why this is happening? Is this because filebeats is still holding onto the deleted files and causing read and resulting in events not sent? or perhaps this is something else?

Events pattern showing the drop. The solid part is when I restarted all the filebeats collector and log generation. The spikes happens on the hour.

Config below:

filebeat.spool_size: 16384
filebeat.prospectors:

- input_type: log

  paths:
- /logs/current/*.log
  document_type: data
  json.keys_under_root: true
  fields:
     customkey: uuid12345
  fields_under_root: true
  scan_frequency: 1s
  close_inactive: 2m
  close_renamed: true

#----------------------------- Logstash output --------------------------------
output.logstash:
  hosts: ["OUR_COLLECTRS"]
  loadbalance: true
  ssl.enabled: true
  ssl.certificate_authorities: ["OUR CERT"]
  ssl.verification_mode: none
  compression_level: 3
  bulk_max_size: 2048

#logging.level: debug
logging.selectors: ["*"]

additional data:
the log files in the folder will first get renamed, then archived to a different folder then deleted. this happens hourly (matching the pattern)

Is there any issues on the filebeat side? or perhaps my logstashes aren't keeping up. I am seeing high CPU utilization on the logstash side of things.

This is a sample of what I am seeing once the rotation is done.

This topic was automatically closed after 21 days. New replies are no longer allowed.