Harvesting logs from many docker containers

Elastic Stack Beats
1

I think I might have gone down the wrong path here so just need a little guidance and some pros/cons to what I've done vs other ways of solving this problem.

I have multiple docker containers each with a microservice in, and more will come in time. Each logs out in various different formats, apache, gunicorn, custom nlog, custom python logs etc.

Right now I'm exposing the log files from each container with docker volume mappings in the docker-componse.yml e.g.:

services:
  vehicle_service:
    image: ...
    volumes:
      - ../logs/microservice1/:/root/microservce1/logs

Then filebeat in my logging stack (4 ELKB docker containers) is configured like:

services:
  filebeat:
    image: ...
    volumes:
      - ./filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml
      - ../deployment/logs:/usr/share/logs/deployment

with the filebeat.yml:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/share/logs/deployment/**/*.log
  tags: ["microservice"]

I can use different subfolders under logs to filter out different types of logs and tag them for different parsing in logstash.

This all works fine.

BUT, is this a good way to do it? It seems wrong to have this file location dependency between the microservice containers and my logging stack containers. And I need to think about where this common folder location is... if my services are distributed on different nodes I need network storage.

I've seen other examples using the docker type and a filebeat prospector. Is this a better way for my scenario?

Thanks

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.