Filebeat 5.5.1 is not work with symlink file when symlinks option has been set true

Elastic Stack Beats
1
  • Version: 5.5.1
  • Operating System: Linux
  • Steps to Reproduce:

I use it in kubernetes.
I try to use it as a sidecar in order to forward logs from the containers in same pod.

apiVersion: v1
kind: ConfigMap
metadata:
  name: counter-filebeat-config
  labels:
    app: counter
data:
  filebeat.yml: |
    filebeat.prospectors:
    - input_type: log
      paths:
        - /var/log/containers/${POD_NAME}_*.log
      exclude_files:
        - filebeat
      symlinks: true
    output.console:
      pretty: true
      enabled: true


---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: counter
  labels:
    app: counter
spec:
  replicas: 1
  template:
    metadata:
      namespace: default
      labels:
        app: counter
    spec:
      restartPolicy: Always
      containers:
      - image: busybox
        name: counter
        args: [/bin/sh, -c,
                'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done']
        imagePullPolicy: IfNotPresent
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:5.5.1
        command: ["filebeat", "-e", "-d", "*"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        volumeMounts:
          - name: beat-config
            mountPath: /usr/share/filebeat/filebeat.yml
            subPath: filebeat.yml
          - name: containers-log
            mountPath: /var/log/containers
      volumes:
        - name: containers-log
          hostPath: 
            path: /var/log/containers
        - name: beat-config
          configMap:
            name: counter-filebeat-config

Filebeat logs:

2017-08-14T14:23:32.522152669Z 2017/08/14 14:23:32.521898 prospector_log.go:70: DBG  Start next scan
2017-08-14T14:23:32.52215814Z 2017/08/14 14:23:32.521983 prospector_log.go:161: DBG  stat(/var/log/containers/counter-3417886863-bjnwv_default_counter-14af587b340a7c482099871e03814f3afdc60aee6e8b110db6b4cd611599ec51.log) failed: stat /var/log/containers/counter-3417886863-bjnwv_default_counter-14af587b340a7c482099871e03814f3afdc60aee6e8b110db6b4cd611599ec51.log: no such file or directory
2017-08-14T14:23:32.522163517Z 2017/08/14 14:23:32.521998 prospector_log.go:136: DBG  Exclude file: /var/log/containers/counter-3417886863-bjnwv_default_filebeat-61195818d2c937527f4d9523da34d676072b345052c7bc6da6f5dfb9d0d4b25a.log
2017-08-14T14:23:32.522167701Z 2017/08/14 14:23:32.522006 prospector_log.go:91: DBG  Prospector states cleaned up. Before: 0, After: 0
2017-08-14T14:23:32.539145701Z 2017/08/14 14:23:32.538973 spooler.go:89: DBG  Flushing spooler because of timeout. Events flushed: 0
2017-08-14T14:23:37.539378989Z 2017/08/14 14:23:37.539121 spooler.go:89: DBG  Flushing spooler because of timeout. Events flushed: 0
2017-08-14T14:23:42.522366123Z 2017/08/14 14:23:42.522122 prospector.go:183: DBG  Run prospector
2017-08-14T14:23:42.52238635Z 2017/08/14 14:23:42.522150 prospector_log.go:70: DBG  Start next scan
2

Oh, I think it maybe because the default user of the docker image used to run filebeat is not root.

3

Sorry, It's my fault.

There are two reason.

  • First:
    The default user of the docker image used to run filebeat is not root.
    Resolve it by building myself image just change the user to root.
FROM docker.elastic.co/beats/filebeat:5.5.1
USER root
  • Second:
    After follow the symlink 2 time, the real file is in path "/var/lib/docker/containers/[container-uuid]/[random id].log". The path(/var/lib/docker/containers/) must be mounted in as a volume.

The Final k8s yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: counter-filebeat-config
  labels:
    app: counter
data:
  filebeat.yml: |
    filebeat.prospectors:
    - input_type: log
      paths:
        - /var/log/containers/${POD_NAME}_*.log
      exclude_files:
        - filebeat
      symlinks: true
    output.console:
      pretty: true
      enabled: true


---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: counter
  labels:
    app: counter
spec:
  replicas: 1
  template:
    metadata:
      namespace: default
      labels:
        app: counter
    spec:
      restartPolicy: Always
      containers:
      - image: busybox
        name: counter
        args: [/bin/sh, -c,
                'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done']
        imagePullPolicy: IfNotPresent
      - name: filebeat
        image: sdcx/filebeat:5.5.1
        command: ["filebeat", "-e", "-d", "*"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        volumeMounts:
          - name: beat-config
            mountPath: /usr/share/filebeat/filebeat.yml
            subPath: filebeat.yml
          - name: docker-log
            mountPath: /var/lib/docker/containers/
          - name: containers-log
            mountPath: /var/log
      volumes:
        - name: docker-log
          hostPath:
            path: /var/lib/docker/containers/
        - name: containers-log
          hostPath: 
            path: /var/log
        - name: beat-config
          configMap:
            name: counter-filebeat-config
2 Likes
4

This topic was automatically closed after 21 days. New replies are no longer allowed.