[Filebeat] aws-cloudwatch input sends logs only after the filebeat service restarts

Filebeats ver: 7.13
I am using the aws-cloudwatch plugin with the following input configuration.

[2021-07-10 13:39:49.883Z:30560:I:__main__:wst-infra:39] Invoking: sshi on account: beacon-test-tko region: None
filebeat.inputs:

- type: aws-cloudwatch
  enabled: true
  log_group_name: /aws/rds/cluster/sdlc-cluster/postgresql
  region_name: ap-northeast-1
  start_position: beginning
  fields:
    service_name: aws_aurora

...

output.file:
  path: /tmp/filebeat
  filename: filebeat
  number_of_files: 10

The service is running on an EC2 instances which has the appropriate role and access to cloudwatch logs. I run a few test scripts that create ERROR logs for the configured log group. However, I can only see the logs once the filebeat service is restarted/reloaded. I previously had this input under inputs.d/cloudwatch.yml and could only see my output file update when I made a change on this input file. Seems like only restarting and reloading makes filebeat ship the logs. I've also tried a longer api_sleep by setting it to 2 seconds. Still no luck.

Logs don't tell much:

2021-07-10T14:49:10.350Z	INFO	[registrar]	registrar/registrar.go:109	States Loaded from registrar: 20
2021-07-10T14:49:10.350Z	INFO	[crawler]	beater/crawler.go:71	Loading Inputs: 1
2021-07-10T14:49:10.350Z	WARN	[cfgwarn]	awscloudwatch/input.go:82	BETA: aws-clouwatch input type is used
2021-07-10T14:49:10.350Z	INFO	[aws-cloudwatch]	awscloudwatch/input.go:128	Initialized AWS CloudWatch input.
2021-07-10T14:49:10.350Z	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 12693341086000574317)
2021-07-10T14:49:10.351Z	INFO	[crawler]	beater/crawler.go:108	Loading and starting Inputs completed. Enabled inputs: 1
2021-07-10T14:49:10.351Z	INFO	[aws-cloudwatch]	awscloudwatch/input.go:137	aws-cloudwatch input worker for log group: '/aws/rds/cluster/sdlc-cluster/postgresql' has started
2021-07-10T14:49:10.352Z	INFO	cfgfile/reload.go:164	Config reloader started
2021-07-10T14:49:10.352Z	INFO	cfgfile/reload.go:224	Loading of config files completed.
2021-07-10T14:49:40.004Z	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"filebeat.service"},"cpuacct":{"id":"filebeat.service","total":{"ns":704874666}},"memory":{"id":"filebeat.service","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":33173504}}}},"cpu":{"system":{"ticks":50,"time":{"ms":55}},"total":{"ticks":690,"time":{"ms":703},"value":690},"user":{"ticks":640,"time":{"ms":648}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":13},"info":{"ephemeral_id":"b9de2507-49e4-4aef-8804-924bf1e55287","uptime":{"ms":30130}},"memstats":{"gc_next":24351808,"memory_alloc":18872688,"memory_sys":75514880,"memory_total":102080008,"rss":80449536},"runtime":{"goroutines":23}},"filebeat":{"events":{"added":1366,"done":1366},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"acked":1366,"active":0,"batches":1,"total":1366},"type":"file","write":{"bytes":1441909}},"pipeline":{"clients":1,"events":{"active":0,"published":1366,"total":1366},"queue":{"acked":1366}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"load":{"1":0,"15":0.15,"5":0,"norm":{"1":0,"15":0.075,"5":0}}}}}}
2021-07-10T14:50:10.004Z	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":17769637}},"memory":{"mem":{"usage":{"bytes":200704}}}},"cpu":{"system":{"ticks":50,"time":{"ms":4}},"total":{"ticks":710,"time":{"ms":18},"value":710},"user":{"ticks":660,"time":{"ms":14}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":12},"info":{"ephemeral_id":"b9de2507-49e4-4aef-8804-924bf1e55287","uptime":{"ms":60130}},"memstats":{"gc_next":24351808,"memory_alloc":19770464,"memory_total":102977784,"rss":80728064},"runtime":{"goroutines":21}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0,"15":0.14,"5":0,"norm":{"1":0,"15":0.07,"5":0}}}}}}```