Scaling filebeat over containers


I’m looking for the appropriate way to monitor applicative logs produced nginx, tomcat, springboot embedded in docker with filebeat and ELK.

In the container strategy, a container should be use for only one purpose

  • one nginx per container
  • one tomcat per container
    meaning we can’t have an additional filebeat within a nginx or tomcat container.

Over what I have read over Internet, we could have the following setup:

  • a volume dedicated for storing logs
  • a nginx container which mount the dedicated logs volume
  • a tomcat / springboot container which mount the dedicated logs volume
  • a filebeat container also mounting the dedicated logs volume

This works fine but when it comes to scale out nginx and springboot container, it is a little bit more complex for me
Which pattern should I use to push my logs using filebeat to logstash if I have the following configuration:

  • several nginx containers in load balancing with the same configuration (logs configuration is the same: same path)
  • several springboot rest api containers behing nginx containers with the same configuration (logs configuration is the same:same path)

Should I create one volume by set of nginx + springboot rest api and add a filebeat container ?
Should I create a global log volume shared by all my containers and have a different log filename by container
(having the name of the container in the filename of the logs?) and having only one filebeat container ?
In the second proposal, how to scale filebeat ?
Is there another way to do that ?

Many thanks for your help.

I believe the most common approach is to have a global shared log volume, and have one Filebeat pointing at all of it. The bottleneck in Filebeat is usually its connection to the Elasticsearch server, and running multiple Filebeats on the same machine won't increase that bandwidth much in typical configurations. I'd usually only try multiple Filebeats if you're considering sending the different log types to different elasticsearch endpoints.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.