Metricbeat on AWS ECS using Fargate

Elastic Stack Beats
1

I am trying to run Metricbeat using docker as a "sidecar" container in an ECS Fargate Task. Unfortunately when running inside Fargate containers cannot mount volumes on the host such as the /var/run/docker.sock required by metricbeat.

What is the recommended way to ship logs and metrics from my ECS containers to Elastic Search?

1 Like
2

Hello, I don't think this is required by metricbeat. In Run Metricbeat on Docker | Metricbeat Reference [8.11] | Elastic you can see, this is configurable so maybe give this a try?

3

Thanks for the feedback. Without the above mount how will metricbeat get access to the docker metrics?

4

@Zachary_Baxter I have been striking out trying to get some help on this topic, was hoping you might be able to point me in the right direction.

I’m trying to get container metrics and logs shipped to elastic. Problem is these containers are running in AWS ECS fargate.
ECS Fargate doesn't allow host volume mounting, so we cannot mount /var/run/docker.sock or /var/lib/docker/containers. This means filebeat and metrcibeat cannot get access to the data needed to ship to elastic.

I tried to find the recommended approach to shipping logs and metrics to elasticsearch when using ECS with fargate, I found nothing.

To better understand things I know that a similar system Datadog provides what we are looking for. Turns out this system was a launch partner with AWS fargate. They get Metrics by having their agent interact with the local AWS endpoint and ship data https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v2.html. They get Logs by introducing a lambda that collects the logs from cloud watch and then ships the data. More details https://docs.datadoghq.com/integrations/ecs_fargate/

Before I go off and create some custom beat or the overhead of passing things through cloudwatch and lambda I wanted to see what the experts recommended.

5

For any folks using Fargate... The feedback I am getting from the team at Elastic is to use Functionbeat for logs and then to follow the same technique Data Dog uses (https://docs.datadoghq.com/integrations/ecs_fargate/).

I was able to setup functionbeat to get my container logs shipped to elastic from fargate. This is working really great with some reasonable latencies. I'm seeing 1-5 second latencies between log time and ingest time with the default settings.

Next I am going to try building a custom beat that runs in a sidecar container and publishes metrics found in https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v2.html up to elastic. If I can publish these in the same format that metricbeat for docker does that will be great.

I will continue to post my progress on this thread, just in case it helps any other fargate users out there.

3 Likes
6

Hey Monster ant progress towards this effort, i'm facing the same dilemma, maybe we can work on this together? let me know.

7

@masterxavierfox unfortunately I was pulled off onto a higher priority task. I probably wont be back on this for a few weeks. But I'd love to share resources, I will definitely put anything I come up with on github so we can collaborate on it.

2 Likes
8

Great! looking forward to it @monster

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.