I am using filebeat docker image (8.7.1) to run an ecs service that reads cloudwatch logs and send to an index. My cloudwatch log groups gets created dynamically so i am using "log_group_name_prefix" to identify all log groups matching certain prefix like "/aws/ecs/iv1/runs". I am having 2 issues with the "log_group_name_prefix" configuration which i am detailing below.
- Filebeat is not reading incremental log groups.
Lets assume when i started the ecs filebeat service i had 50 log groups matching the prefix criteria, then only thoes 50 log groups are being scanned for log streams based on the no of workers and scan frequency given. The new log groups that gets added after start of filebeat/ecs service is not getting scanned for log streams. I can see when i restart the service the new log groups are also scanned.
This is the config i used
-type:aws-cloudwatch
log_group_name_prefix: /aws/ecs/iv1/runs
region_name: eu-west-1
number_of_workers: 10
scan_frequency: 1m
start_position: beginning
- Issue with scan frequency, no of workers and start position
Here in the same case of having 50 log groups, when the filebeat service start it takes 10 log groups in initial scan ( equivalent to number of workers) and for these 10 log groups i can see that the start time is 1970-01-01 and end time is current time. So this is scanning all log streams in the considered log groups. However in the next scan ( log groups 11 to 20) it is considering start time as 1 minute from current time. In this case though there are log streams in the considered log groups, none of the are being send to index as the time duration may not be matching.
Please confirm on the above two behavior, if it is as it is expected to be. Also please do suggest what is the ideal way to deal with dynamic log groups/log streams using cloud-watch input.