Metricbeat on AWS ECS using Fargate


(jacob locken) #1

I am trying to run Metricbeat using docker as a "sidecar" container in an ECS Fargate Task. Unfortunately when running inside Fargate containers cannot mount volumes on the host such as the /var/run/docker.sock required by metricbeat.

What is the recommended way to ship logs and metrics from my ECS containers to Elastic Search?


(Kaiyan Sheng) #2

Hello, I don't think this is required by metricbeat. In https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-docker.html#_volume_mounted_configuration you can see, this is configurable so maybe give this a try?


(jacob locken) #3

Thanks for the feedback. Without the above mount how will metricbeat get access to the docker metrics?


(jacob locken) #4

@Zachary_Baxter I have been striking out trying to get some help on this topic, was hoping you might be able to point me in the right direction.

I’m trying to get container metrics and logs shipped to elastic. Problem is these containers are running in AWS ECS fargate.
ECS Fargate doesn't allow host volume mounting, so we cannot mount /var/run/docker.sock or /var/lib/docker/containers. This means filebeat and metrcibeat cannot get access to the data needed to ship to elastic.

I tried to find the recommended approach to shipping logs and metrics to elasticsearch when using ECS with fargate, I found nothing.

To better understand things I know that a similar system Datadog provides what we are looking for. Turns out this system was a launch partner with AWS fargate. They get Metrics by having their agent interact with the local AWS endpoint and ship data https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v2.html. They get Logs by introducing a lambda that collects the logs from cloud watch and then ships the data. More details https://docs.datadoghq.com/integrations/ecs_fargate/

Before I go off and create some custom beat or the overhead of passing things through cloudwatch and lambda I wanted to see what the experts recommended.


(jacob locken) #5

For any folks using Fargate... The feedback I am getting from the team at Elastic is to use Functionbeat for logs and then to follow the same technique Data Dog uses (https://docs.datadoghq.com/integrations/ecs_fargate/).

I was able to setup functionbeat to get my container logs shipped to elastic from fargate. This is working really great with some reasonable latencies. I'm seeing 1-5 second latencies between log time and ingest time with the default settings.

Next I am going to try building a custom beat that runs in a sidecar container and publishes metrics found in https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v2.html up to elastic. If I can publish these in the same format that metricbeat for docker does that will be great.

I will continue to post my progress on this thread, just in case it helps any other fargate users out there.