Hello!
I've been trying to poll for Cloudwatch metrics from AWS ECS Fargate to our Elastic Cloud to have everything centralized in terms of alerting and monitoring. However, with the current configuration, no indices are being created in Elasticsearch. Please see config below:
input {
cloudwatch {
namespace => "AWS/ECS"
metrics => ["CPUUtilization", "MemoryUtilization"]
filters => {"ClusterName" => "main-cluster"}
period => 60
region => "ap-southeast-1"
}
cloudwatch {
namespace => "AWS/ECS"
metrics => ["CPUUtilization", "MemoryUtilization"]
filters => {"ClusterName" => "services-cluster"}
period => 60
region => "ap-southeast-1"
}
}
filter {
}
output {
elasticsearch {
hosts => ["<ES_CLOUD_URL>"]
index => "cloudwatch-metrics-%{+YYYY.MM.dd}"
user => "user"
password => "pass"
}
stdout {}
}
There are no errors shown in Logstash as well. The logs say only this:
[2019-07-29T07:51:35,004][INFO ][logstash.inputs.cloudwatch] Polling CloudWatch API
[2019-07-29T07:51:35,032][INFO ][logstash.inputs.cloudwatch] Polling CloudWatch API
[2019-07-29T07:51:35,038][INFO ][logstash.inputs.cloudwatch] [Aws::CloudWatch::Client 200 0.031198 0 retries] get_metric_statistics(namespace:"AWS/ECS",metric_name:"MemoryUtilization",start_time:2019-07-29 07:36:35 UTC,end_time:2019-07-29 07:51:35 UTC,period:60,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"ClusterName",value:"[FILTERED]"}])
[2019-07-29T07:51:35,053][INFO ][logstash.inputs.cloudwatch] [Aws::CloudWatch::Client 200 0.018497 0 retries] get_metric_statistics(namespace:"AWS/ECS",metric_name:"CPUUtilization",start_time:2019-07-29 07:36:35 UTC,end_time:2019-07-29 07:51:35 UTC,period:60,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"ClusterName",value:"[FILTERED]"}])
[2019-07-29T07:51:35,074][INFO ][logstash.inputs.cloudwatch] [Aws::CloudWatch::Client 200 0.01739 0 retries] get_metric_statistics(namespace:"AWS/ECS",metric_name:"MemoryUtilization",start_time:2019-07-29 07:36:35 UTC,end_time:2019-07-29 07:51:35 UTC,period:60,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"ClusterName",value:"[FILTERED]"}])
[2019-07-29T07:51:35,082][INFO ][logstash.inputs.cloudwatch] [Aws::CloudWatch::Client 200 0.034519 0 retries] get_metric_statistics(namespace:"AWS/ECS",metric_name:"CPUUtilization",start_time:2019-07-29 07:36:35 UTC,end_time:2019-07-29 07:51:35 UTC,period:60,statistics:["SampleCount","Average","Minimum","Maximum","Sum"],dimensions:[{name:"ClusterName",value:"[FILTERED]"}])
Any advise is greatly appreciated!