@Zachary_Baxter I have been striking out trying to get some help on this topic, was hoping you might be able to point me in the right direction.
I’m trying to get container metrics and logs shipped to elastic. Problem is these containers are running in AWS ECS fargate.
ECS Fargate doesn't allow host volume mounting, so we cannot mount
/var/lib/docker/containers. This means filebeat and metrcibeat cannot get access to the data needed to ship to elastic.
I tried to find the recommended approach to shipping logs and metrics to elasticsearch when using ECS with fargate, I found nothing.
To better understand things I know that a similar system Datadog provides what we are looking for. Turns out this system was a launch partner with AWS fargate. They get Metrics by having their agent interact with the local AWS endpoint and ship data https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v2.html. They get Logs by introducing a lambda that collects the logs from cloud watch and then ships the data. More details https://docs.datadoghq.com/integrations/ecs_fargate/
Before I go off and create some custom beat or the overhead of passing things through cloudwatch and lambda I wanted to see what the experts recommended.