# \[Filebeat\] aws-cloudwatch input sends logs only after the filebeat service restarts **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [July 10, 2021, 2:41pm UTC](https://discuss.elastic.co/t/filebeat-aws-cloudwatch-input-sends-logs-only-after-the-filebeat-service-restarts/278336 "2021-07-10T14:41:48Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![Luci](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luci/32/91469_2.png) [@Luci](https://discuss.elastic.co/u/Luci)\ **Post date:** [July 10, 2021, 2:41pm UTC](https://discuss.elastic.co/t/filebeat-aws-cloudwatch-input-sends-logs-only-after-the-filebeat-service-restarts/278336/1 "2021-07-10T14:41:48Z")
Filebeats ver: 7.13 I am using the `aws-cloudwatch` plugin with the following input configuration. ```auto [2021-07-10 13:39:49.883Z:30560:I: __main__ :wst-infra:39] Invoking: sshi on account: beacon-test-tko region: None filebeat.inputs: - type: aws-cloudwatch enabled: true log_group_name: /aws/rds/cluster/sdlc-cluster/postgresql region_name: ap-northeast-1 start_position: beginning fields: service_name: aws_aurora ... output.file: path: /tmp/filebeat filename: filebeat number_of_files: 10 ``` The service is running on an EC2 instances which has the appropriate role and access to cloudwatch logs. I run a few test scripts that create ERROR logs for the configured log group. However, I can only see the logs once the filebeat service is restarted/reloaded. I previously had this input under inputs.d/cloudwatch.yml and could only see my output file update when I made a change on this input file. Seems like only restarting and reloading makes filebeat ship the logs. I've also tried a longer api\_sleep by setting it to 2 seconds. Still no luck. Logs don't tell much: ````auto 2021-07-10T14:49:10.350Z INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 20 2021-07-10T14:49:10.350Z INFO [crawler] beater/crawler.go:71 Loading Inputs: 1 2021-07-10T14:49:10.350Z WARN [cfgwarn] awscloudwatch/input.go:82 BETA: aws-clouwatch input type is used 2021-07-10T14:49:10.350Z INFO [aws-cloudwatch] awscloudwatch/input.go:128 Initialized AWS CloudWatch input. 2021-07-10T14:49:10.350Z INFO [crawler] beater/crawler.go:141 Starting input (ID: 12693341086000574317) 2021-07-10T14:49:10.351Z INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 1 2021-07-10T14:49:10.351Z INFO [aws-cloudwatch] awscloudwatch/input.go:137 aws-cloudwatch input worker for log group: '/aws/rds/cluster/sdlc-cluster/postgresql' has started 2021-07-10T14:49:10.352Z INFO cfgfile/reload.go:164 Config reloader started 2021-07-10T14:49:10.352Z INFO cfgfile/reload.go:224 Loading of config files completed. 2021-07-10T14:49:40.004Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"filebeat.service"},"cpuacct":{"id":"filebeat.service","total":{"ns":704874666}},"memory":{"id":"filebeat.service","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":33173504}}}},"cpu":{"system":{"ticks":50,"time":{"ms":55}},"total":{"ticks":690,"time":{"ms":703},"value":690},"user":{"ticks":640,"time":{"ms":648}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":13},"info":{"ephemeral_id":"b9de2507-49e4-4aef-8804-924bf1e55287","uptime":{"ms":30130}},"memstats":{"gc_next":24351808,"memory_alloc":18872688,"memory_sys":75514880,"memory_total":102080008,"rss":80449536},"runtime":{"goroutines":23}},"filebeat":{"events":{"added":1366,"done":1366},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"acked":1366,"active":0,"batches":1,"total":1366},"type":"file","write":{"bytes":1441909}},"pipeline":{"clients":1,"events":{"active":0,"published":1366,"total":1366},"queue":{"acked":1366}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"load":{"1":0,"15":0.15,"5":0,"norm":{"1":0,"15":0.075,"5":0}}}}}} 2021-07-10T14:50:10.004Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":17769637}},"memory":{"mem":{"usage":{"bytes":200704}}}},"cpu":{"system":{"ticks":50,"time":{"ms":4}},"total":{"ticks":710,"time":{"ms":18},"value":710},"user":{"ticks":660,"time":{"ms":14}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":12},"info":{"ephemeral_id":"b9de2507-49e4-4aef-8804-924bf1e55287","uptime":{"ms":60130}},"memstats":{"gc_next":24351808,"memory_alloc":19770464,"memory_total":102977784,"rss":80728064},"runtime":{"goroutines":21}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0,"15":0.14,"5":0,"norm":{"1":0,"15":0.07,"5":0}}}}}}``` ```` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 7, 2021, 4:41pm UTC](https://discuss.elastic.co/t/filebeat-aws-cloudwatch-input-sends-logs-only-after-the-filebeat-service-restarts/278336/2 "2021-08-07T16:41:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.