Filebeats ver: 7.13
I am using the aws-cloudwatch plugin with the following input configuration.
[2021-07-10 13:39:49.883Z:30560:I:__main__:wst-infra:39] Invoking: sshi on account: beacon-test-tko region: None
filebeat.inputs:
- type: aws-cloudwatch
enabled: true
log_group_name: /aws/rds/cluster/sdlc-cluster/postgresql
region_name: ap-northeast-1
start_position: beginning
fields:
service_name: aws_aurora
...
output.file:
path: /tmp/filebeat
filename: filebeat
number_of_files: 10
The service is running on an EC2 instances which has the appropriate role and access to cloudwatch logs. I run a few test scripts that create ERROR logs for the configured log group. However, I can only see the logs once the filebeat service is restarted/reloaded. I previously had this input under inputs.d/cloudwatch.yml and could only see my output file update when I made a change on this input file. Seems like only restarting and reloading makes filebeat ship the logs. I've also tried a longer api_sleep by setting it to 2 seconds. Still no luck.
Logs don't tell much:
2021-07-10T14:49:10.350Z INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 20
2021-07-10T14:49:10.350Z INFO [crawler] beater/crawler.go:71 Loading Inputs: 1
2021-07-10T14:49:10.350Z WARN [cfgwarn] awscloudwatch/input.go:82 BETA: aws-clouwatch input type is used
2021-07-10T14:49:10.350Z INFO [aws-cloudwatch] awscloudwatch/input.go:128 Initialized AWS CloudWatch input.
2021-07-10T14:49:10.350Z INFO [crawler] beater/crawler.go:141 Starting input (ID: 12693341086000574317)
2021-07-10T14:49:10.351Z INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 1
2021-07-10T14:49:10.351Z INFO [aws-cloudwatch] awscloudwatch/input.go:137 aws-cloudwatch input worker for log group: '/aws/rds/cluster/sdlc-cluster/postgresql' has started
2021-07-10T14:49:10.352Z INFO cfgfile/reload.go:164 Config reloader started
2021-07-10T14:49:10.352Z INFO cfgfile/reload.go:224 Loading of config files completed.
2021-07-10T14:49:40.004Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"filebeat.service"},"cpuacct":{"id":"filebeat.service","total":{"ns":704874666}},"memory":{"id":"filebeat.service","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":33173504}}}},"cpu":{"system":{"ticks":50,"time":{"ms":55}},"total":{"ticks":690,"time":{"ms":703},"value":690},"user":{"ticks":640,"time":{"ms":648}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":13},"info":{"ephemeral_id":"b9de2507-49e4-4aef-8804-924bf1e55287","uptime":{"ms":30130}},"memstats":{"gc_next":24351808,"memory_alloc":18872688,"memory_sys":75514880,"memory_total":102080008,"rss":80449536},"runtime":{"goroutines":23}},"filebeat":{"events":{"added":1366,"done":1366},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"acked":1366,"active":0,"batches":1,"total":1366},"type":"file","write":{"bytes":1441909}},"pipeline":{"clients":1,"events":{"active":0,"published":1366,"total":1366},"queue":{"acked":1366}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"load":{"1":0,"15":0.15,"5":0,"norm":{"1":0,"15":0.075,"5":0}}}}}}
2021-07-10T14:50:10.004Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":17769637}},"memory":{"mem":{"usage":{"bytes":200704}}}},"cpu":{"system":{"ticks":50,"time":{"ms":4}},"total":{"ticks":710,"time":{"ms":18},"value":710},"user":{"ticks":660,"time":{"ms":14}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":12},"info":{"ephemeral_id":"b9de2507-49e4-4aef-8804-924bf1e55287","uptime":{"ms":60130}},"memstats":{"gc_next":24351808,"memory_alloc":19770464,"memory_total":102977784,"rss":80728064},"runtime":{"goroutines":21}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0,"15":0.14,"5":0,"norm":{"1":0,"15":0.07,"5":0}}}}}}```