Filebeat installed as Daemon Set in IBM Private Cloud Kubernetes Cluster don't send logs to Elastic Cloud instance

[l.rava (lorenzo rava)]

Hi
I installed as Daemon Set metricBeat and fileBeat into my infrastracture:

• IBM Private Cloud
• kubernetes cluster - 2 nodes - v.1.13
  following this guide.

MetricBeat send data without problem. Filebeat don't send data.

here logs in debug for filebeat:

INFO instance/beat.go:571 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
DEBUG [beat] instance/beat.go:623 Beat metadata path: /usr/share/filebeat/data/meta.json
INFO instance/beat.go:579 Beat ID: 8498bd87-16c3-42a3-b5e0-4ca23133a429
INFO [index-management.ilm] ilm/ilm.go:129 Policy name: filebeat-7.1.1
DEBUG [filters] add_cloud_metadata/add_cloud_metadata.go:164 add_cloud_metadata: starting to fetch metadata, timeout=3s
DEBUG [filters] add_cloud_metadata/add_cloud_metadata.go:196 add_cloud_metadata: received disposition for qcloud after 339.455729ms. result=[provider:qcloud, error=failed requesting qcloud metadata: Get http://metadata.tencentyun.com/meta-data/instance-id: dial tcp: lookup metadata.tencentyun.com on 172.21.0.10:53: no such host, metadata={}]
DEBUG [filters] add_cloud_metadata/add_cloud_metadata.go:196 add_cloud_metadata: received disposition for openstack after 3.000222149s. result=[provider:openstack, error=failed requesting openstack metadata: Get http://169.254.169.254/2009-04-04/meta-data/hostname: dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
DEBUG [filters] add_cloud_metadata/add_cloud_metadata.go:203 add_cloud_metadata: timed-out waiting for all responses
DEBUG [filters] add_cloud_metadata/add_cloud_metadata.go:167 add_cloud_metadata: fetchMetadata ran for 3.000347097s
INFO add_cloud_metadata/add_cloud_metadata.go:346 add_cloud_metadata: hosting provider type not detected.
DEBUG [processors] processors/processor.go:66 Processors: add_cloud_metadata=null
DEBUG [seccomp] seccomp/seccomp.go:109 Loading syscall filter {"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4",...,"writev"],"action":"allow"}]}}}
INFO [seccomp] seccomp/seccomp.go:116 Syscall filter successfully installed
INFO [beat] instance/beat.go:827 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "8498bd87-16c3-42a3-b5e0-4ca23133a429"}}}
INFO [beat] instance/beat.go:836 Build info {"system_info": {"build": {"commit": "3358d9a5a09e3c6709a2d3aaafde628ea34e8419", "libbeat": "7.1.1", "time": "2019-05-23T13:21:33.000Z", "version": "7.1.1"}}}
INFO [beat] instance/beat.go:839 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.11.5"}}}
INFO [beat] instance/beat.go:843 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-04-26T14:06:48Z","containerized":false,"name":"kube-fra02-cr7a*************49fc787c-w5.cloud.ibm","ip":["127.0.0.1/8","172.20.0.1/32","::1/128","10.XX.225.43/26","fe80::467:43ff:fec8:f89c/64","158.XX.138.101/28","158.XX.145.222/32","fe80::xxx:72ff:fe53:6022/64","127.0.0.10/31","fe80::bcaa:c3ff:fe81:c38/64","172.30.7.64/32"}}}
INFO [beat] instance/beat.go:872 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter"}, "start_time": "2019-06-05T12:35:27.460Z"}}}
INFO instance/beat.go:280 Setup Beat: filebeat; Version: 7.1.1
DEBUG [beat] instance/beat.go:301 Initializing output plugins
INFO [index-management] idxmgmt/std.go:165 Set output.elasticsearch.index to 'filebeat-7.1.1' as ILM is enabled.
INFO elasticsearch/client.go:165 Elasticsearch url: https://51fxxxxxxxxxxxxf3c92201363c.europe-west1.gcp.cloud.es.io:443
DEBUG [publisher] pipeline/consumer.go:137 start pipeline event consumer
INFO [publisher] pipeline/module.go:97 Beat name: kube-fra02-cr7a4aeac0xxxxxxx40c8c9799d049fc787c-w5.cloud.ibm
INFO instance/beat.go:391 filebeat start running.
DEBUG [test] registrar/migrate.go:159 isFile(/usr/share/filebeat/data/registry) -> false
DEBUG [test] registrar/migrate.go:159 isFile() -> false
DEBUG [test] registrar/migrate.go:152 isDir(/usr/share/filebeat/data/registry/filebeat) -> true
DEBUG [test] registrar/migrate.go:159 isFile(/usr/share/filebeat/data/registry/filebeat/meta.json) -> true
INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
DEBUG [registrar] registrar/migrate.go:51 Registry type '0' found
DEBUG [registrar] registrar/registrar.go:125 Registry file set to: /usr/share/filebeat/data/registry/filebeat/data.json
INFO registrar/registrar.go:145 Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
INFO registrar/registrar.go:152 States Loaded from registrar: 0
INFO crawler/crawler.go:72 Loading Inputs: 0
INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 0
WARN [cfgwarn] kubernetes/kubernetes.go:55 BETA: The kubernetes autodiscover is beta
INFO kubernetes/util.go:86 kubernetes: Using pod name kube-fra02-cr7a4aeacxxxxx799d049fc787c-w5.cloud.ibm and namespace kube-system to discover kubernetes node
INFO cfgfile/reload.go:150 Config reloader started
DEBUG [registrar] registrar/registrar.go:278 Starting Registrar
ERROR kubernetes/util.go:90 kubernetes: Querying for pod failed with error: kubernetes api: Failure 404 pods "kube-fra02-cr7a4aeac036c7440c8c9799d049fc787c-w5.cloud.ibm" not found

[l.rava (lorenzo rava)]

deploy yaml content:

@l.rava

deploy yaml content:

apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.config:
##inputs:
# Mounted filebeat-inputs configmap:
## path: ${path.config}/inputs.d/.yml
# Reload inputs configs as they change:
## reload.enabled: false
modules:
path: ${path.config}/modules.d/.yml
# Reload module configs as they change:
reload.enabled: true

# To enable hints based autodiscover, remove `filebeat.config.inputs` configuration and uncomment this:
filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true

processors:
  - add_cloud_metadata:

cloud.id: ${ELASTIC_CLOUD_ID}
cloud.auth: ${ELASTIC_CLOUD_AUTH}

output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}

apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-inputs
namespace: kube-system
labels:
k8s-app: filebeat
data:
kubernetes.yml: |-
- type: docker
containers.ids:
- "*"
processors:
- add_kubernetes_metadata:
in_cluster: true

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: filebeat
namespace: kube-system
labels:
k8s-app: filebeat
spec:
template:
metadata:
labels:
k8s-app: filebeat
spec:
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat:7.1.1
args: [
"-c", "/etc/filebeat.yml",
"-e",
"-d",
"*",
"-E",
""cloud.id=elk-log-stack:ZX2EyYjc5YQ=="",
"-E",
""cloud.auth=elastic:Hc""
]
env:
- name: ELASTIC_CLOUD_ID
value: "elk-log-stack:ZX2EyYjc5YQ=="
- name: ELASTIC_CLOUD_AUTH
value: "elastic:Hc"
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
runAsUser: 0
# If using Red Hat OpenShift uncomment this:
#privileged: true
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
- name: inputs
mountPath: /usr/share/filebeat/inputs.d
readOnly: true
- name: data
mountPath: /usr/share/filebeat/data
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
volumes:
- name: config
configMap:
defaultMode: 0600
name: filebeat-config
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: inputs
configMap:
defaultMode: 0600
name: filebeat-inputs
# data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
- name: data
hostPath:
path: /var/lib/filebeat-data
type: DirectoryOrCreate

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:

• kind: ServiceAccount
  name: filebeat
  namespace: kube-system
  roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: filebeat
labels:
k8s-app: filebeat
rules:

• apiGroups: [""] # "" indicates the core API group
  resources:
  • namespaces
  • pods
    verbs:
  • get
  • watch
  • list

---

apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: kube-system
labels:
k8s-app: filebeat

[exekias (Carlos Pérez Aradros)]

Hi @l.rava,

Thank you for your feedback! Our default manifests expect logs under /var/lib/docker/containers, as it's the most common scenario, I'm wondering if that's the case for IBM Cloud. Could you please ssh one of the nodes and check how the logs are stored?

You can go to /var/log/pods and check log files there. They may be common files or symlinks to some other folder.

[Ben_Stucke]

@l.rava Have you solved the problem. I have have the same problem. I am using filebeat 7.15.0 and used the guide on Run Filebeat on Kubernetes | Filebeat Reference [7.15] | Elastic.
My kubernetes Version on IBM Cloud is 1.19.14_1557. I have checked that my logs on the kubernetes node are stored on "/var/log/containers". If i go into the filebeat container to the the corresponding path all log files are there too. Heartbeat and Metricbeat works fine and filebeat works with this configuration on minikube. I don't see any errors but my Elasticsearch instance on elastic.co does not have any logs.

[Ben_Stucke]

Setting the volumeMounts and volumes solved it for me:

     volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log/containers
          readOnly: true
        - name: varlogpods
          mountPath: /var/log/pods
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0640
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log/containers
      - name: varlogpods
        hostPath:
          path: /var/log/pods
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          # When filebeat runs as non-root user, this directory needs to be writable by group (g+w).
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate