Kubernetes & Docker max-size

Elastic Stack Beats
1

Hi! We're trying to better manage our Kubernetes logs. I have Filebeat running in the cluster using autodiscover and shipping off to Logstash.

Say in the K8s cluster, I have the Docker logging driver set to max-size of 10MB. When that file size is reached - Docker rolls the log.

Filebeat will pick up the new log, correct?

In Docker, if I set max-file to like 2, this will help ensure that Filebeat will still be able to finish reading from the old log file and then go on to the new. And so forth and so forth as logs are rotated.

That sound about right?

Thanks!

2

That sounds about right.

The part I'm not sure anymore on how the docker log driver exactly rotates files. Can you share some details on this?

3

Cool beans.

So in the Docker daemon configuration, you set the logging driver to JSON (the default anyway) and can set max-size and max-files options:

 {
   "log-driver": "json-file",
   "log-opts": {
     "max-size": "10m",
     "max-file": "5"
   }
 }

The only draw back, is looking at the Kubernetes logging documentation (Logging Architecture | Kubernetes), if you have more than one log file, it seems like kubectl logs will output the latest:

Note: Currently, if some external system has performed the rotation, only the contents of the latest log file will be available through kubectl logs. E.g. if there’s a 10MB file, logrotate performs the rotation and there are two files, one 10MB in size and one empty, kubectl logs will return an empty response.

Which really shouldn't be a problem if Filebeat is working correctly as we should be getting all output into Elastic and view-able in Kibana.

4

Can you share the config you are using? Also a log file would be useful best on debug level to see what files Filebeat opens and closes.

What is the naming pattern the rotated files have?

5

Thanks! I'm running a test with Docker now so will have a naming pattern shortly.

Here's our configuration:

 data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          in_cluster: true
          templates:
            - condition:
                or:
                  - equals:
                      kubernetes.namespace: development
                  - equals:
                      kubernetes.namespace: sqa
                  - equals:
                      kubernetes.namespace: test
                  - equals:
                      kubernetes.namespace: stage
                  - equals:
                      kubernetes.namespace: elasticsearch-curator
              config:
                - type: docker
                  containers.ids:
                    - "${data.kubernetes.container.id}"
                  multiline.pattern: '^[[:space:]]'
                  multiline.negate: false
                  multiline.match: after

    output.logstash:
      hosts: ['${LOGSTASH_HOST}:${LOGSTASH_PORT}']
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-inputs
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  kubernetes.yml: |-
    - type: docker
      containers.ids:
      - "*"
      processors:
        - add_kubernetes_metadata:
            in_cluster: true

I guess my main concern is with autodiscover, Filebeat is hitting the K8s API for pod creation / removal, correct? Will Filebeat still pickup log file rotation changes with autodiscover?

Thanks!

-Matt

6

Here's how Docker rotates the logs:

7f362ab3750980588f111c848382db52a9ef3e6c9c89f96e0fd5de037a82792d-json.log
7f362ab3750980588f111c848382db52a9ef3e6c9c89f96e0fd5de037a82792d-json.log.1
7f362ab3750980588f111c848382db52a9ef3e6c9c89f96e0fd5de037a82792d-json.log.2
7f362ab3750980588f111c848382db52a9ef3e6c9c89f96e0fd5de037a82792d-json.log.3
7f362ab3750980588f111c848382db52a9ef3e6c9c89f96e0fd5de037a82792d-json.log.4

7

docker input will always read from the .log file, including when it's created after rotation.

I recently discovered it will be better sense to put a pattern for the rotated files in the list of paths, that will ensure Filebeat gets those too in case it's down for a while.

Best regards

8

But then I can't limit based on namespace right? I don't want to pickup CNI crap or anything else Kubernetes is throwing under the hood just yet. I just want our application logs.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.