NGINX-Filebeat-ELK-Docker-Swarm Help

[JockDaRock (Jock Reed)]

Hello, I am looking to deploy docker swarm services for filebeat and the ELK stack to process NGINX gateway access logs. The architecture would be nginx, filebeat, elasticsearch, kibana, and logstash as seperate services and sharing volumes where necessary to process NGINX logs and display metrics from the nginx gateway.

Kubernetes seems a bit more straightforward to me than Docker swarm on the config yaml and I have not been able to get a successful test up in Docker Swarm.

Does anyone have an example of this working in Docker swarm or is anyone here experienced with Docker Swarm configurations to make this work.

Please let me know your thoughts.

[iremmats (Mats Iremark)]

We are using nginx and filebeat together on several places. Here is a start for you

  nginx:
    image: nginx
    configs:
     - source: nginx_config
       target: /etc/nginx/nginx.conf
    volumes:
    - nginx_logs:/var/log/nginx/
    deploy:
      mode: global
  filebeat-nginx:
    image: docker.elastic.co/beats/filebeat:6.2.2
    volumes:
    - nginx_logs:/var/log/nginx/
    environment:
    - ELASTIC_URL=${ELASTIC_URL}
    - ELASTIC_USERNAME=${ELASTIC_FILEBEAT_USERNAME}
    - ELASTIC_PASSWORD=${ELASTIC_FILEBEAT_PASSWORD}
    configs:
    - source: filebeat_nginx_config
      target: /usr/share/filebeat/filebeat.yml
      mode: 0600
      uid: '1000'
      gid: '1000'
    deploy:
      mode: global
volumes:
  nginx_logs:
configs:
  nginx_config:
    file: ./src/nginx/nginx.conf

Since nginx per default doesnt write its logs to file when inside a container we added these to nginx.conf.

access_log /var/log/nginx/access2.log;
error_log /var/log/nginx/error2.log;

And filebeat.yml contains this

filebeat.modules:
- module: nginx
  access:
    enabled: true
    var.paths: ["/var/log/nginx/access2.log"]
  error:
    enabled: true
    var.paths: ["/var/log/nginx/error2.log"]

In our case filebeat sends to a Redis cache and then Logstash processes from there.
Hope it helps!