Hi everyone! After a lot of work i managed to create the ELK in my system with everything (Kibana, Elasticsearch, Logstash, Metricbeat) and i am moving to the next step which is monitoring the apache logs from a Apache/PHP container. As a extra context, i have a Docker Swarm with a Manager and 2 Workers with both workers in Drain state for now. My issue is that i am creating a container with Filebeat that collects the log information from the containers in the host and i have a LAMP service that has a Apache-PHP container and a MariaDB container linked to each other. But even when i am monitoring the docker folder on the host, FIlebeat is not getting any apache metrics. My config files are as follows
Filebeat service creation
docker service create \ --name filebeat \ --user=root \ --mount type=bind,src=/home/repositories/ELK/filebeat/filebeat_2.yml,dst=/usr/share/filebeat/filebeat.yml \ --mount type=bind,src=/home/repositories/ELK/filebeat/apache.yml,dst=/usr/share/filebeat/modules.d/apache.yml \ --mount type=bind,src=/var/lib/docker/containers,dst=/var/lib/docker/containers,readonly \ --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock,readonly \ --network elk_stack \ --mode global \ docker.elastic.co/beats/filebeat:7.3.2
filebeat_2.yml
filebeat.config: modules: path: ${path.config}/modules.d/*.yml reload.enabled: false filebeat.autodiscover: providers: - type: docker hints.enabled: true processors: - add_docker_metadata: ~ output.logstash: hosts: ["logstash:5044"] setup.kibana: host: "kibana:5601" #setup.dashboards.enabled: true
apache.yml
# Module: apache # Docs: https://www.elastic.co/guide/en/beats/filebeat/7.3/filebeat-module-apache.html - module: apache # Access logs access: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. var.paths: ["/var/lib/docker/containers/*/*.log"] # Error logs error: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. var.paths: ["/var/lib/docker/containers/*/*.log"]
I see the module enable
Enabled:
apacheDisabled:
apache
auditd
cisco
coredns
elasticsearch
envoyproxy
googlecloud
haproxy
icinga
iis
iptables
kafka
kibana
logstash
mongodb
mssql
mysql
nats
netflow
nginx
osquery
panw
postgresql
rabbitmq
redis
santa
suricata
system
traefik
zeek
Apache service logs
centrodecontactos_poc.1.gzo8etdq00m6@MITLLDKR01 | 10.255.0.2 - - [26/Sep/2019:19:28:21 +0000] "GET /favicon.ico HTTP/1.1" 200 1718 "http://172.31.28.105:8081/?page=guest&action=index&correo=sdasdada" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36" centrodecontactos_poc.1.gzo8etdq00m6@MITLLDKR01 | 10.255.0.2 - - [26/Sep/2019:19:28:21 +0000] "GET /images/logoHelper.png HTTP/1.1" 304 192 "http://172.31.28.105:8081/?page=guest&action=index&correo=sdasdada" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36" centrodecontactos_poc.1.gzo8etdq00m6@MITLLDKR01 | 10.255.0.2 - - [26/Sep/2019:19:28:21 +0000] "GET /?page=guest&action=index&correo=sdasdada HTTP/1.1" 200 1305 "http://172.31.28.105:8081/?page=guest&action=index&correo=sdaad" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36" centrodecontactos_poc.1.gzo8etdq00m6@MITLLDKR01 | 10.255.0.2 - - [26/Sep/2019:19:28:21 +0000] "GET /images/logoHelper.png HTTP/1.1" 304 192 "http://172.31.28.105:8081/?page=guest&action=index&correo=sdasdada" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36" centrodecontactos_poc.1.gzo8etdq00m6@MITLLDKR01 | 10.255.0.2 - - [26/Sep/2019:19:28:21 +0000] "GET /?page=guest&action=index&correo=sdasdada HTTP/1.1" 200 1305 "http://172.31.28.105:8081/?page=guest&action=index&correo=sdaad" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36" centrodecontactos_poc.1.gzo8etdq00m6@MITLLDKR01 | 10.255.0.2 - - [26/Sep/2019:19:28:21 +0000] "GET /images/logoHelper.png HTTP/1.1" 304 192 "http://172.31.28.105:8081/?page=guest&action=index&correo=sdasdada" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36" centrodecontactos_poc.1.gzo8etdq00m6@MITLLDKR01 | 10.255.0.2 - - [26/Sep/2019:19:28:21 +0000] "GET /?page=guest&action=index&correo=sdasdada HTTP/1.1" 200 1305 "http://172.31.28.105:8081/?page=guest&action=index&correo=sdaad" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36" centrodecontactos_poc.1.gzo8etdq00m6@MITLLDKR01 | 10.255.0.2 - - [26/Sep/2019:19:28:21 +0000] "GET /images/logoHelper.png HTTP/1.1" 304 192 "http://172.31.28.105:8081/?page=guest&action=index&correo=sdasdada" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36" centrodecontactos_poc.1.gzo8etdq00m6@MITLLDKR01 | 10.255.0.2 - - [26/Sep/2019:19:28:21 +0000] "GET /?page=guest&action=index&correo=sdasdada HTTP/1.1" 200 1305 "http://172.31.28.105:8081/?page=guest&action=index&correo=sdaad" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36" centrodecontactos_poc.1.gzo8etdq00m6@MITLLDKR01 | 10.255.0.2 - - [26/Sep/2019:19:28:21 +0000] "GET /images/logoHelper.png HTTP/1.1" 304 192 "http://172.31.28.105:8081/?page=guest&action=index&correo=sdasdada" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36" centrodecontactos_poc.1.gzo8etdq00m6@MITLLDKR01 | 10.255.0.2 - - [26/Sep/2019:19:28:22 +0000] "GET /favicon.ico HTTP/1.1" 200 1717 "http://172.31.28.105:8081/?page=guest&action=index&correo=sdasdada" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36"