We are using filebeats to ingest log files that are rotated hourly. This is sent to a load-balanced between 4 hosts running logstash behind HA proxy. I have been tweaking with the close_* options as I am seeing a huge build up of "deleted" files that are held by filebeats, eventhough the log files are cycled. As a result, I am seeing the follow pattern in our data, where on top of the hour, there's an influx of data, the events will start to drop off until the next hour. If I disable filebeats and the logging for a period of time, the data will flow solid for a while then the "drops" and pattern starts again. Any insight on why this is happening? Is this because filebeats is still holding onto the deleted files and causing read and resulting in events not sent? or perhaps this is something else?
Events pattern showing the drop. The solid part is when I restarted all the filebeats collector and log generation. The spikes happens on the hour.
filebeat.spool_size: 16384 filebeat.prospectors: - input_type: log paths: - /logs/current/*.log document_type: data json.keys_under_root: true fields: customkey: uuid12345 fields_under_root: true scan_frequency: 1s close_inactive: 2m close_renamed: true #----------------------------- Logstash output -------------------------------- output.logstash: hosts: ["OUR_COLLECTRS"] loadbalance: true ssl.enabled: true ssl.certificate_authorities: ["OUR CERT"] ssl.verification_mode: none compression_level: 3 bulk_max_size: 2048 #logging.level: debug logging.selectors: ["*"]
the log files in the folder will first get renamed, then archived to a different folder then deleted. this happens hourly (matching the pattern)
Is there any issues on the filebeat side? or perhaps my logstashes aren't keeping up. I am seeing high CPU utilization on the logstash side of things.
This is a sample of what I am seeing once the rotation is done.