filtering setup for docker containers not working.
I started Elastic Stack without anybody additional settings. And here are my filebeat settings:
filebeat.yml
filebeat.autodiscover:
providers:
- type: docker
labels.dedot: true
hints.enabled: true
templates:
- condition:
contains:
container.labels.collect_logs_with_filebeat: "true" # label name in service docker-compose file
docker.container.name: "test_golang_app_2"
config:
- type: container
format: docker # auto, docker, cli
# stream: stdout # all, stdout, stderr
#containers.ids:
# - "${data.docker.container.id}"
paths:
- "/var/lib/docker/containers/${data.docker.containers.id}/*.log"
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.kibana:
host: "localhost:5601"
output.elasticsearch:
enabled: true
hosts: ["localhost:9200"]
output.logstash:
enabled: false
hosts: ["localhost:5044"]
processors:
- drop_fields:
fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.name", "agent.version", "docker.container.labels.com_docker_compose_config-hash", "docker.container.labels.com_docker_compose_container-number", "docker.contain>
ignore_missing: false
monitoring.enabled: false
logging.metrics.enabled: false
logging.level: debug
logging.selectors: ["*"]
logging.to_files: true
and to test, I wrote a simple application in docker container that constantly sends data to the stdout stream
docker-compose.yml
version: '3.7'
services:
simple_golang_app:
image: simple_golang_app
container_name: simple_golang_app
build:
context: app/golang/
dockerfile: Dockerfile
args:
TEST_ENV: $TEST_ENV
networks:
- net
test_golang_app_2:
image: test_golang_app_2
container_name: test_golang_app_2
build:
context: app/golang/
dockerfile: Dockerfile
args:
TEST_ENV: $TEST_ENV
networks:
- net
deploy:
labels:
docker.container.labels.description: "collect_logs_with_filebeat"
co.elastic.logs/enabled: "true" # for Filebeat
collect_logs_with_filebeat: "true"
labels:
docker.container.labels.description: "collect_logs_with_filebeat"
co.elastic.logs/enabled: "true" # for Filebeat
collect_logs_with_filebeat: "true"
networks:
net:
driver: overlay
main.go
package main
import (
"fmt"
"io"
"os"
"time" // https://pkg.go.dev/time
)
func main() {
fmt.Println("Print from the Go program")
fmt.Println(os.Getenv("TEST_ENV"))
io.WriteString(os.Stdout,"This is the line to standard output.\n")
io.WriteString(os.Stderr,"This is the line for standard error output.\n")
// print every 5 seconds how long the program is running
for range time.Tick(time.Second * 30) {
go func() {
fmt.Println(os.Stdout, time.Now())
}()
}
}
Dockerfile
FROM golang:1.17-alpine
ADD main.go /home
WORKDIR /home
RUN \
apk add --no-cache bash git openssh && \
go get -u github.com/minio/minio-go
CMD ["go","run","main.go"]
The problem is that the templates filter doesn't work. I have tried various methods: by container name, by adding labels, but I still see the logs of all running containers on my host, and not a specific one.
How can I still configure the sending of logs for certain containers, and not all those running on the host?