How fast are files created.
it also depends on the loads. It is hard for me to give you exact numbers. In a problematic VM, we could have a new file almost every minute but with little data inside (~20MB).
How fast are files rotated (that is, how long until they reach 200M)
Well, it varies. In some cases it could take around 3-4 hours.
Your current config.
Here is our current config
filebeat:
prospectors:
-
document_type: jobA
input_type: log
paths:
- "/path/to/jobA/logs/*/*.log"
multiline:
pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}'
negate: true
match: after
fields:
instance_name: ""
instance_id: ""
-
document_type: jobB
input_type: log
paths:
- "/path/to/jobB/logs/*/*/std*"
- "/path/to/jobB/logs2/*/std*"
multiline:
pattern: '(^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3})|(^[0-9]{2}/[0-9]{2}/[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2})'
negate: true
match: after
fields:
instance_name: ""
instance_id: ""
registry_file: /path/to/filebeat/registry
output:
logstash:
hosts: [":5044"]
bulk_max_size: 1024
shipper:
logging:
to_syslog: false
to_files: true
files:
path: /path/to/filebeat/log
name: filebeat.log
rotateeverybytes: 10485760 # = 10MB
keepfiles: 7