Filebeat not harvesting anything in Kubernetes


#1

New to filebeats. I had a working configuration on docker, trying to get a simple config working in Kubernetes.

Using the deploy sample from 6.6 branch

I don't seem to be able to get Filebeats to parse logs or do anything. My output file never gets created. I don't see anything in the logs about "harvesting"

I've tried both with autodiscover and inputs

Here is my Kubernetes Manifests.

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: test
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
     providers:
       - type: kubernetes
         hints.enabled: true
    processors:
      - add_cloud_metadata:
    output.file:
     path: "/filebeat"
     filename: "filebeat.log"
     rotate_every_kb: 10000
     number_of_files: 7
     permissions: 0600
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-inputs
  namespace: test
  labels:
    k8s-app: filebeat
data:
  kubernetes.yml: |-
    - type: docker
      containers.ids:
      - "*"
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: test
  labels:
    k8s-app: filebeat
spec:
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:6.6.2
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value: elastic
        - name: ELASTICSEARCH_PASSWORD
          value: changeme
        - name: ELASTIC_CLOUD_ID
          value:
        - name: ELASTIC_CLOUD_AUTH
          value:
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: inputs
          mountPath: /usr/share/filebeat/inputs.d
          readOnly: true
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: inputs
        configMap:
          defaultMode: 0600
          name: filebeat-inputs
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: test
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: test
  labels:
    k8s-app: filebeat
---

#2

Sample log I get

 2019-04-16T20:00:01.870Z	INFO	instance/beat.go:616	Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2019-04-16T20:00:01.870Z	INFO	instance/beat.go:623	Beat UUID: f722eebb-b2cc-4c3f-a358-fcb695b2e10d
2019-04-16T20:00:01.871Z	INFO	[seccomp]	seccomp/seccomp.go:116	Syscall filter successfully installed
2019-04-16T20:00:01.871Z	INFO	[beat]	instance/beat.go:936	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "f722eebb-b2cc-4c3f-a358-fcb695b2e10d"}}}
2019-04-16T20:00:01.871Z	INFO	[beat]	instance/beat.go:945	Build info	{"system_info": {"build": {"commit": "1eea934ce81be553337f2828bd12131896fea8e4", "libbeat": "6.6.2", "time": "2019-03-06T14:17:59.000Z", "version": "6.6.2"}}}
2019-04-16T20:00:01.871Z	INFO	[beat]	instance/beat.go:948	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.10.8"}}}
2019-04-16T20:00:01.873Z	INFO	[beat]	instance/beat.go:952	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-03-29T15:20:27Z","containerized":true,"name":"filebeat-njth7","ip":["127.0.0.1/8","172.16.49.25/32"],"kernel_version":"4.14.104-95.84.amzn2.x86_64","mac":["32:9b:75:17:4c:a2"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2019-04-16T20:00:01.873Z	INFO	[beat]	instance/beat.go:981	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2019-04-16T20:00:00.750Z"}}}
2019-04-16T20:00:01.874Z	INFO	instance/beat.go:281	Setup Beat: filebeat; Version: 6.6.2
2019-04-16T20:00:04.874Z	INFO	add_cloud_metadata/add_cloud_metadata.go:319	add_cloud_metadata: hosting provider type not detected.
2019-04-16T20:00:04.875Z	INFO	elasticsearch/client.go:165	Elasticsearch url: https://redacted:443
2019-04-16T20:00:04.875Z	INFO	[publisher]	pipeline/module.go:110	Beat name: filebeat-njth7
2019-04-16T20:00:04.875Z	INFO	instance/beat.go:403	filebeat start running.
2019-04-16T20:00:04.876Z	INFO	registrar/registrar.go:134	Loading registrar data from /usr/share/filebeat/data/registry
2019-04-16T20:00:04.876Z	INFO	registrar/registrar.go:141	States Loaded from registrar: 0
2019-04-16T20:00:04.876Z	INFO	crawler/crawler.go:72	Loading Inputs: 0
2019-04-16T20:00:04.876Z	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 0
2019-04-16T20:00:04.876Z	WARN	[cfgwarn]	kubernetes/kubernetes.go:55	BETA: The kubernetes autodiscover is beta
2019-04-16T20:00:04.876Z	INFO	kubernetes/util.go:86	kubernetes: Using pod name filebeat-njth7 and namespace test to discover kubernetes node
2019-04-16T20:00:04.877Z	INFO	[monitoring]	log/log.go:117	Starting metrics logging every 30s
2019-04-16T20:00:04.896Z	INFO	kubernetes/util.go:93	kubernetes: Using node ip-redacted.ec2.internal discovered by in cluster pod node query
2019-04-16T20:00:04.896Z	WARN	[cfgwarn]	hints/logs.go:56	BETA: The hints builder is beta
2019-04-16T20:00:04.896Z	INFO	autodiscover/autodiscover.go:104	Starting autodiscover manager
2019-04-16T20:00:04.897Z	INFO	kubernetes/watcher.go:182	kubernetes: Performing a resource sync for *v1.PodList
2019-04-16T20:00:04.903Z	INFO	log/input.go:138	Configured paths: [/var/lib/docker/containers/b978909fa8f52801678d9671db05f1326e5f4229198d89da95d0c347035f3f8d/*.log]
2019-04-16T20:00:04.903Z	INFO	input/input.go:114	Starting input of type: docker; ID: 12223340593314259418 
2019-04-16T20:00:04.903Z	INFO	log/input.go:138	Configured paths: [/var/lib/docker/containers/2ded579b5f294b443a3d048bfa43eb94f79add681fad5b19af77e788fd811a9d/*.log]
2019-04-16T20:00:04.904Z	INFO	input/input.go:114	Starting input of type: docker; ID: 9406250489401606417 
2019-04-16T20:00:04.904Z	INFO	log/input.go:138	Configured paths: [/var/lib/docker/containers/8bfe628080440dda5e4a21453ffd3fe8ee218ffa35c728c2fec3a1e19fe667df/*.log]
2019-04-16T20:00:04.904Z	INFO	input/input.go:114	Starting input of type: docker; ID: 17620474456461264372 
2019-04-16T20:00:04.905Z	INFO	log/input.go:138	Configured paths: [/var/lib/docker/containers/0d057dc69462d6884ae6b548ddae0f082d9e3183ad11efc9b199cd0d9c7dba7f/*.log]
2019-04-16T20:00:04.905Z	INFO	input/input.go:114	Starting input of type: docker; ID: 2204726011095176077 
2019-04-16T20:00:04.905Z	INFO	kubernetes/watcher.go:198	kubernetes: Resource sync done
2019-04-16T20:00:04.905Z	INFO	kubernetes/watcher.go:242	kubernetes: Watching API for resource events
2019-04-16T20:00:04.905Z	INFO	log/input.go:138	Configured paths: [/var/lib/docker/containers/0d09b0b546712ef8ffbf609d06d56722ad83d044e2d6a7deb28559adac5af914/*.log]
2019-04-16T20:00:04.905Z	INFO	input/input.go:114	Starting input of type: docker; ID: 13749717482665840743 
2019-04-16T20:00:04.906Z	INFO	log/input.go:138	Configured paths: [/var/lib/docker/containers/8e6c54ff5a8aabf7aff3653101b00ede96bea8e8846cca5c183f4d6b2a3bbb4f/*.log]
2019-04-16T20:00:04.906Z	INFO	input/input.go:114	Starting input of type: docker; ID: 15176759354301759303 

#3

Not sure if this means anything, but if I exec into the filebeat container, I see nothing in /var/lib/docker/containers


(Jaime Soriano) #4

Hi @bbgobie,

Are you using docker as container runtime for Kubernetes?
Do you see anything in /var/lib/docker/containers in the host?


#5

Thanks Jaime,
We are using docker, checked with admin, and our docker logs don't actually go to /var/lib/docker/containers.
Adjusted the mountpath and Filebeat is harvesting!