We are using filebeats to ingest log files that are rotated hourly. This is sent to a load-balanced between 4 hosts running logstash behind HA proxy. I have been tweaking with the close_* options as I am seeing a huge build up of "deleted" files that are held by filebeats, eventhough the log files are cycled. As a result, I am seeing the follow pattern in our data, where on top of the hour, there's an influx of data, the events will start to drop off until the next hour. If I disable filebeats and the logging for a period of time, the data will flow solid for a while then the "drops" and pattern starts again. Any insight on why this is happening? Is this because filebeats is still holding onto the deleted files and causing read and resulting in events not sent? or perhaps this is something else?
Events pattern showing the drop. The solid part is when I restarted all the filebeats collector and log generation. The spikes happens on the hour.
Config below:
filebeat.spool_size: 16384
filebeat.prospectors:
- input_type: log
paths:
- /logs/current/*.log
document_type: data
json.keys_under_root: true
fields:
customkey: uuid12345
fields_under_root: true
scan_frequency: 1s
close_inactive: 2m
close_renamed: true
#----------------------------- Logstash output --------------------------------
output.logstash:
hosts: ["OUR_COLLECTRS"]
loadbalance: true
ssl.enabled: true
ssl.certificate_authorities: ["OUR CERT"]
ssl.verification_mode: none
compression_level: 3
bulk_max_size: 2048
#logging.level: debug
logging.selectors: ["*"]
additional data:
the log files in the folder will first get renamed, then archived to a different folder then deleted. this happens hourly (matching the pattern)
Is there any issues on the filebeat side? or perhaps my logstashes aren't keeping up. I am seeing high CPU utilization on the logstash side of things.
This is a sample of what I am seeing once the rotation is done.
