Filebeat Not Resuming Rotated File(s)


#1

Lots of messages (765 so far today) with the following in my filebeats log file:

filebeat version 1.3.0 (amd64)
Description: Oracle Linux Server release 6.7
VM with 4vCPU and 8G RAM

Four prospectors with 2-9 individual log files to watch setup pretty much like:

[code] -
paths:
- /var/storage/central-log-store/192.168.1.1.log
- /var/storage/central-log-store/192.168.1.2.log
- /var/storage/central-log-store/192.168.1.3.log
input_type: log
include_lines: [
"ASA-3-109032",
"ASA-5-750006",
"ASA-5-751025",
"ASA-6-113004",
"ASA-6-113005",
"ASA-6-113008",
"ASA-6-113009",
"ASA-6-113010",
"ASA-6-113039",
"ASA-6-302014",
"ASA-6-302016",
"ASA-6-602303",
"ASA-6-725001",
"ASA-6-725002",
"ASA-6-751025",
"ASA-6-751026",
"ASA-3-109032",
"ASA-4-113019",
"ASA-4-113030",
"ASA-4-113033",
"ASA-4-113034",
"ASA-5-109029"
]
exclude_files: [".gz$"]
fields:
timezone: "America/New_York"
fields_under_root: true
document_type: cisco-fw

registry_file: /var/lib/filebeat/registry

output:
logstash:
hosts: [ "192.168.2.1:5044" ]
#worker: 1
compression_level: 3 #gzip compression level - effects host CPU
[/code]

Some of these log files grow to about 1 to 1.5G every hour or so when they get rotated to a renamed file such as 192.168.1.1.log.1 and then on the second rotation get compressed with gzip.

Trying to figure out what's the base issue here. Is the file not being picked up when it is being rotated? Is Filebeat not able to keep up with all the lines out of a log before it's rotated? Host is maxed out?


(ruflin) #2

The log message here is not ideal. It means that a file was rotated but now need data was added, so no new harvester has to be started.

All seems to be fine as long as you don't miss any events.

Side note: I strongly recommend to update to 1.3.1 or 5.0-beta1 because of a known bug in 1.3.0.


(system) #3

This topic was automatically closed after 21 days. New replies are no longer allowed.