How to support AWS cloudwatch metric stream to Elasticsearch?

zhaoyi0113 · July 26, 2021, 10:14pm

I am deploying applications to AWS including various services. I am planning to use Kibana and Elasticsearch to view metrics from Cloudwatch. I know I can configure metricbeat by using AWS module which works as a crawler to pull metrics data from AWS to Elasticsearch. But there is another mode which is to use Cloudwatch metric stream to stream metics data to analytics platform.

I wonder how does metricbeat support stream data to Elasticsearch. Is there a metrics server I can use to listen on the metric data stream?

legoguy1000 · July 27, 2021, 1:21am

What do u mean exactly? Are you asking if the data from the cloudwatch metricbeat module can be sent somewhere besides elasticsearch?

stephenb · July 27, 2021, 2:57am

Hi @zhaoyi0113 welcome to the community.

Today Metricbeat does not directly support the new Cloudwatch Metric stream.

You could ingest that data with Logstash with the kinesis input.

I do not have insight at this time whether the Cloudwatch Metric stream is on the Metricbeat roadmap. If I find out I will post back here.

zhaoyi0113 · July 27, 2021, 5:03am

I can ingest these data into elasticsearch. How can I make Kibana to recognise the metric data and show them in the metrics view? Is there any data format I should follow?

stephenb · July 27, 2021, 2:04pm

Hi @zhaoyi0113

To be transparent I suspect there would need to be considerable field mapping exercise to do it on your own.

The here is the Elastic Common Schema

My suggestion would to be use the Metricbeat Cloudwatch module first so you could see all the fields etc and then work from there.

system · August 24, 2021, 4:04pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.