How do i get AWS cloudwatch metrics to elastic for serverless

Elastic Stack Elasticsearch
1

Hello,

I am trying to push cloudwatch metrics via sqs to elastic. How do i go about doing this?
these are for multiple servers and serverless applications

2

Hi @rpeynado1 Welcome to the community.

There are a number of ways to ingest cloudwatch metrics via SQS or perhaps other ways.

Important:

  • What version of the Elastic Stack are you on?
  • Self-Managed or Elastic Cloud Hosted, Elastic Cloud Serverless :slight_smile:
  • Do you have Fleet Running?

Elastic Agent is a method you could use. Note for this method you need to provide a VM(s) to run the Elastic Agent on.

There is a blog on this very topic

You can also ingest the metrics directly from Cloudwatch

If you can route your metrics through AWS Firehose you could use this..

4

hi @stephenb

thank you for the response

  1. our version is 8.15.2
  2. we are using ECk
  3. and we do have a fleet running

i will take a look at those articles. the one with firehouse wont work. i think our infrastructure does not allow it

5

Sounds like Elastic Agent with the
SQS Input
Or
Just the direct Cloudwatch Metrics integration
would be a good place to start

Note if you have high volume, you can run more than 1 Agent against an SQS queue for parallelization.

6

@stephenb i do need a server for the elastic agent right? is it possible to use an elastic agent in a serverless environment? forgive me if these are dumb questions i am extremely new to this process

7

Running agent on ECK / K8s can be done but can be tricky for what you are trying to do. The quickstart is geared toward monitoring the K8s and will be a daemonset... not exactly what you want to do.

You would probably want to convert to a deployment but a daemonset might work.

Otherwise

You can provide a VM(s) to run the Elastic Agent on...

You can run it as a Docker container or deploy it on bare metal.

Ahhh I forgot we also have the Elastic Serverless Forwarder ... perhaps that might be a good fit (I am not an expert on that) but look like it might be a good fit.

8

thank you so much! i really appreciate it. i will look into these. the serverless forwarder looks promising, but i noticed that it is more geared towards "logs" do you reckon it will work for metrics?

9

Hmmm ... yeah kinda looks like that...

So Wait... I am confused what are you trying to ingest?
Exactly what metrics?

In the end... It is probably the Simplest... Fastest ...
to Deploy Fleet Managed Elastic Agent on a VM, then you can try all sorts of integration easily... that is what I would do...

Then, I would optimize and scale after you understand what you need, what you are shipping, and what Integration and Processing you need...

In the end you need to roll up your sleeves, pull out the Helmet and Flashlight and give it a try :slight_smile: :flashlight:

10

@stephenb I was able to flush out some more details on the issue. We have an elastic agent for AWS. We are trying to turn on the Collect “SQS metrics” function but its giving a “list metrics error”. Do you have any insights on how to trouble shoot this?

11

Hi @rpeynado1

You are going to need to provide a lot more detail.

SQS Metrics is a lot different than your original post...

Please show the Exact Integration you are trying to configure and any full error messages etc... where they are coming from etc..etc..

Have you upgraded the integration to the latest ? v2.38.1

My initial guess is that you may have a connectivity issue from inside your K8s ECK from the agent container out to the SQS ... that is not based on anything impercial just a guess...

As I said, running elastic agents INSIDE k8s and reaching outside assets can be tricky.

I usually suggest get the elastic agent and integration working on a standalone EC2 instance Then work on moving that configuration into K8S / ECK

That is just a suggestion