How do i get AWS cloudwatch metrics to elastic for serverless

Elastic Stack Elasticsearch
1

Hello,

I am trying to push cloudwatch metrics via sqs to elastic. How do i go about doing this?
these are for multiple servers and serverless applications

2

Hi @rpeynado1 Welcome to the community.

There are a number of ways to ingest cloudwatch metrics via SQS or perhaps other ways.

Important:

  • What version of the Elastic Stack are you on?
  • Self-Managed or Elastic Cloud Hosted, Elastic Cloud Serverless :slight_smile:
  • Do you have Fleet Running?

Elastic Agent is a method you could use. Note for this method you need to provide a VM(s) to run the Elastic Agent on.

There is a blog on this very topic

You can also ingest the metrics directly from Cloudwatch

If you can route your metrics through AWS Firehose you could use this..

4

hi @stephenb

thank you for the response

  1. our version is 8.15.2
  2. we are using ECk
  3. and we do have a fleet running

i will take a look at those articles. the one with firehouse wont work. i think our infrastructure does not allow it

5

Sounds like Elastic Agent with the
SQS Input
Or
Just the direct Cloudwatch Metrics integration
would be a good place to start

Note if you have high volume, you can run more than 1 Agent against an SQS queue for parallelization.

6

@stephenb i do need a server for the elastic agent right? is it possible to use an elastic agent in a serverless environment? forgive me if these are dumb questions i am extremely new to this process

7

Running agent on ECK / K8s can be done but can be tricky for what you are trying to do. The quickstart is geared toward monitoring the K8s and will be a daemonset... not exactly what you want to do.

You would probably want to convert to a deployment but a daemonset might work.

Otherwise

You can provide a VM(s) to run the Elastic Agent on...

You can run it as a Docker container or deploy it on bare metal.

Ahhh I forgot we also have the Elastic Serverless Forwarder ... perhaps that might be a good fit (I am not an expert on that) but look like it might be a good fit.

8

thank you so much! i really appreciate it. i will look into these. the serverless forwarder looks promising, but i noticed that it is more geared towards "logs" do you reckon it will work for metrics?

9

Hmmm ... yeah kinda looks like that...

So Wait... I am confused what are you trying to ingest?
Exactly what metrics?

In the end... It is probably the Simplest... Fastest ...
to Deploy Fleet Managed Elastic Agent on a VM, then you can try all sorts of integration easily... that is what I would do...

Then, I would optimize and scale after you understand what you need, what you are shipping, and what Integration and Processing you need...

In the end you need to roll up your sleeves, pull out the Helmet and Flashlight and give it a try :slight_smile: :flashlight: