# Filebeat installed as Daemon Set in IBM Private Cloud Kubernetes Cluster don't send logs to Elastic Cloud instance **URL:** \ **Category:** Elastic Cloud on Kubernetes (ECK)\ **Created:** [June 5, 2019, 3:09pm UTC](https://discuss.elastic.co/t/filebeat-installed-as-daemon-set-in-ibm-private-cloud-kubernetes-cluster-dont-send-logs-to-elastic-cloud-instance/184406 "2019-06-05T15:09:11Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![l.rava](https://avatars.discourse-cdn.com/v4/letter/l/a8b319/32.png) [@l.rava](https://discuss.elastic.co/u/l.rava)\ **Post date:** [June 5, 2019, 3:09pm UTC](https://discuss.elastic.co/t/filebeat-installed-as-daemon-set-in-ibm-private-cloud-kubernetes-cluster-dont-send-logs-to-elastic-cloud-instance/184406/1 "2019-06-05T15:09:12Z")
Hi I installed as Daemon Set metricBeat and fileBeat into my infrastracture: - IBM Private Cloud - kubernetes cluster - 2 nodes - v.1.13 following this guide. MetricBeat send data without problem. Filebeat don't send data. here logs in debug for filebeat: > INFO instance/beat.go:571 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs] > DEBUG [beat] instance/beat.go:623 Beat metadata path: /usr/share/filebeat/data/meta.json > INFO instance/beat.go:579 Beat ID: 8498bd87-16c3-42a3-b5e0-4ca23133a429 > INFO [index-management.ilm] ilm/ilm.go:129 Policy name: filebeat-7.1.1 > DEBUG [filters] add\_cloud\_metadata/add\_cloud\_metadata.go:164 add\_cloud\_metadata: starting to fetch metadata, timeout=3s > DEBUG [filters] add\_cloud\_metadata/add\_cloud\_metadata.go:196 add\_cloud\_metadata: received disposition for qcloud after 339.455729ms. result=[provider:qcloud, error=failed requesting qcloud metadata: Get [http://metadata.tencentyun.com/meta-data/instance-id:](http://metadata.tencentyun.com/meta-data/instance-id:) dial tcp: lookup [metadata.tencentyun.com](http://metadata.tencentyun.com) on 172.21.0.10:53: no such host, metadata={}] > DEBUG [filters] add\_cloud\_metadata/add\_cloud\_metadata.go:196 add\_cloud\_metadata: received disposition for openstack after 3.000222149s. result=[provider:openstack, error=failed requesting openstack metadata: Get [http://169.254.169.254/2009-04-04/meta-data/hostname:](http://169.254.169.254/2009-04-04/meta-data/hostname:) dial tcp 169.254.169.254:80: i/o timeout, metadata={}] > DEBUG [filters] add\_cloud\_metadata/add\_cloud\_metadata.go:203 add\_cloud\_metadata: timed-out waiting for all responses > DEBUG [filters] add\_cloud\_metadata/add\_cloud\_metadata.go:167 add\_cloud\_metadata: fetchMetadata ran for 3.000347097s > INFO add\_cloud\_metadata/add\_cloud\_metadata.go:346 add\_cloud\_metadata: hosting provider type not detected. > DEBUG [processors] processors/processor.go:66 Processors: add\_cloud\_metadata=null > DEBUG [seccomp] seccomp/seccomp.go:109 Loading syscall filter {"seccomp\_filter": {"no\_new\_privs":true,"flag":"tsync","policy":{"default\_action":"errno","syscalls":[{"names":["accept","accept4",...,"writev"],"action":"allow"}]}}} > INFO [seccomp] seccomp/seccomp.go:116 Syscall filter successfully installed > INFO [beat] instance/beat.go:827 Beat info {"system\_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "8498bd87-16c3-42a3-b5e0-4ca23133a429"}}} > INFO [beat] instance/beat.go:836 Build info {"system\_info": {"build": {"commit": "3358d9a5a09e3c6709a2d3aaafde628ea34e8419", "libbeat": "7.1.1", "time": "2019-05-23T13:21:33.000Z", "version": "7.1.1"}}} > INFO [beat] instance/beat.go:839 Go runtime info {"system\_info": {"go": {"os":"linux","arch":"amd64","max\_procs":4,"version":"go1.11.5"}}} > INFO [beat] instance/beat.go:843 Host info {"system\_info": {"host": {"architecture":"x86\_64","boot\_time":"2019-04-26T14:06:48Z","containerized":false,"name":"kube-fra02-cr7a\*\*\*\*\*\*\*\*\*\*\*\*\*49fc787c-w5.cloud.ibm","ip":["127.0.0.1/8","172.20.0.1/32","::1/128","10.XX.225.43/26","fe80::467:43ff:fec8:f89c/64","158.XX.138.101/28","158.XX.145.222/32","fe80::xxx:72ff:fe53:6022/64","127.0.0.10/31","fe80::bcaa:c3ff:fe81:c38/64","172.30.7.64/32"}}} > INFO [beat] instance/beat.go:872 Process info {"system\_info": {"process": {"capabilities": {"inheritable":["chown","dac\_override","fowner","fsetid","kill","setgid","setuid","setpcap","net\_bind\_service","net\_raw","sys\_chroot","mknod","audit\_write","setfcap"],"permitted":["chown","dac\_override","fowner","fsetid","kill","setgid","setuid","setpcap","net\_bind\_service","net\_raw","sys\_chroot","mknod","audit\_write","setfcap"],"effective":["chown","dac\_override","fowner","fsetid","kill","setgid","setuid","setpcap","net\_bind\_service","net\_raw","sys\_chroot","mknod","audit\_write","setfcap"],"bounding":["chown","dac\_override","fowner","fsetid","kill","setgid","setuid","setpcap","net\_bind\_service","net\_raw","sys\_chroot","mknod","audit\_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter"}, "start\_time": "2019-06-05T12:35:27.460Z"}}} > INFO instance/beat.go:280 Setup Beat: filebeat; Version: 7.1.1 > DEBUG [beat] instance/beat.go:301 Initializing output plugins > INFO [index-management] idxmgmt/std.go:165 Set output.elasticsearch.index to 'filebeat-7.1.1' as ILM is enabled. > INFO elasticsearch/client.go:165 Elasticsearch url: [https://51fxxxxxxxxxxxxf3c92201363c.europe-west1.gcp.cloud.es.io:443](https://51fxxxxxxxxxxxxf3c92201363c.europe-west1.gcp.cloud.es.io:443) > DEBUG [publisher] pipeline/consumer.go:137 start pipeline event consumer > INFO [publisher] pipeline/module.go:97 Beat name: kube-fra02-cr7a4aeac0xxxxxxx40c8c9799d049fc787c-w5.cloud.ibm > INFO instance/beat.go:391 filebeat start running. > DEBUG [test] registrar/migrate.go:159 isFile(/usr/share/filebeat/data/registry) -\> false > DEBUG [test] registrar/migrate.go:159 isFile() -\> false > DEBUG [test] registrar/migrate.go:152 isDir(/usr/share/filebeat/data/registry/filebeat) -\> true > DEBUG [test] registrar/migrate.go:159 isFile(/usr/share/filebeat/data/registry/filebeat/meta.json) -\> true > INFO [monitoring] log/log.go:117 Starting metrics logging every 30s > DEBUG [registrar] registrar/migrate.go:51 Registry type '0' found > DEBUG [registrar] registrar/registrar.go:125 Registry file set to: /usr/share/filebeat/data/registry/filebeat/data.json > INFO registrar/registrar.go:145 Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json > INFO registrar/registrar.go:152 States Loaded from registrar: 0 > INFO crawler/crawler.go:72 Loading Inputs: 0 > INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 0 > WARN [cfgwarn] kubernetes/kubernetes.go:55 BETA: The kubernetes autodiscover is beta > INFO kubernetes/util.go:86 kubernetes: Using pod name kube-fra02-cr7a4aeacxxxxx799d049fc787c-w5.cloud.ibm and namespace kube-system to discover kubernetes node > INFO cfgfile/reload.go:150 Config reloader started > DEBUG [registrar] registrar/registrar.go:278 Starting Registrar > ERROR kubernetes/util.go:90 kubernetes: Querying for pod failed with error: kubernetes api: Failure 404 pods "kube-fra02-cr7a4aeac036c7440c8c9799d049fc787c-w5.cloud.ibm" not found ---
**Author:** ![l.rava](https://avatars.discourse-cdn.com/v4/letter/l/a8b319/32.png) [@l.rava](https://discuss.elastic.co/u/l.rava)\ **Post date:** [June 5, 2019, 3:11pm UTC](https://discuss.elastic.co/t/filebeat-installed-as-daemon-set-in-ibm-private-cloud-kubernetes-cluster-dont-send-logs-to-elastic-cloud-instance/184406/3 "2019-06-05T15:11:58Z")
deploy yaml content: > [@l.rava](#): > > deploy yaml content: > > > apiVersion: v1 > > kind: ConfigMap > > metadata: > > name: filebeat-config > > namespace: kube-system > > labels: > > k8s-app: filebeat > > data: > > filebeat.yml: |- > > filebeat.config: > > ##inputs: > > # Mounted `filebeat-inputs` configmap: > > ## path: ${path.config}/inputs.d/_.yml > > # Reload inputs configs as they change: > > ## reload.enabled: false > > modules: > > path: ${path.config}/modules.d/_.yml > > # Reload module configs as they change: > > reload.enabled: true > > > > ``` > > # To enable hints based autodiscover, remove `filebeat.config.inputs` configuration and uncomment this: > > filebeat.autodiscover: > > providers: > > - type: kubernetes > > hints.enabled: true > > > > processors: > > - add_cloud_metadata: > > > > cloud.id: ${ELASTIC_CLOUD_ID} > > cloud.auth: ${ELASTIC_CLOUD_AUTH} > > > > output.elasticsearch: > > hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}'] > > username: ${ELASTICSEARCH_USERNAME} > > password: ${ELASTICSEARCH_PASSWORD} > > > > ``` > > > apiVersion: v1 > > kind: ConfigMap > > metadata: > > name: filebeat-inputs > > namespace: kube-system > > labels: > > k8s-app: filebeat > > data: > > kubernetes.yml: |- > > - type: docker > > containers.ids: > > - "\*" > > processors: > > - add\_kubernetes\_metadata: > > in\_cluster: true > > > apiVersion: extensions/v1beta1 > > kind: DaemonSet > > metadata: > > name: filebeat > > namespace: kube-system > > labels: > > k8s-app: filebeat > > spec: > > template: > > metadata: > > labels: > > k8s-app: filebeat > > spec: > > serviceAccountName: filebeat > > terminationGracePeriodSeconds: 30 > > hostNetwork: true > > dnsPolicy: ClusterFirstWithHostNet > > containers: > > - name: filebeat > > image: [docker.elastic.co/beats/filebeat:7.1.1](http://docker.elastic.co/beats/filebeat:7.1.1) > > args: [ > > "-c", "/etc/filebeat.yml", > > "-e", > > "-d", > > "\*", > > "-E", > > ""cloud.id=elk-log-stack:ZX**2EyYjc5YQ=="", > > "-E", > > ""cloud.auth=elastic:H**c"" > > ] > > env: > > - name: ELASTIC\_CLOUD\_ID > > value: "elk-log-stack:ZX**2EyYjc5YQ==" > > - name: ELASTIC\_CLOUD\_AUTH > > value: "elastic:H**c" > > - name: NODE\_NAME > > valueFrom: > > fieldRef: > > fieldPath: spec.nodeName > > securityContext: > > runAsUser: 0 > > # If using Red Hat OpenShift uncomment this: > > #privileged: true > > resources: > > limits: > > memory: 200Mi > > requests: > > cpu: 100m > > memory: 100Mi > > volumeMounts: > > - name: config > > mountPath: /etc/filebeat.yml > > readOnly: true > > subPath: filebeat.yml > > - name: inputs > > mountPath: /usr/share/filebeat/inputs.d > > readOnly: true > > - name: data > > mountPath: /usr/share/filebeat/data > > - name: varlibdockercontainers > > mountPath: /var/lib/docker/containers > > readOnly: true > > volumes: > > - name: config > > configMap: > > defaultMode: 0600 > > name: filebeat-config > > - name: varlibdockercontainers > > hostPath: > > path: /var/lib/docker/containers > > - name: inputs > > configMap: > > defaultMode: 0600 > > name: filebeat-inputs > > # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart > > - name: data > > hostPath: > > path: /var/lib/filebeat-data > > type: DirectoryOrCreate > > > apiVersion: [rbac.authorization.k8s.io/v1beta1](http://rbac.authorization.k8s.io/v1beta1) > > kind: ClusterRoleBinding > > metadata: > > name: filebeat > > subjects: > > > > - kind: ServiceAccount > > name: filebeat > > namespace: kube-system > > roleRef: > > kind: ClusterRole > > name: filebeat > > apiGroup: [rbac.authorization.k8s.io](http://rbac.authorization.k8s.io) > > > > * * * > > > > apiVersion: [rbac.authorization.k8s.io/v1beta1](http://rbac.authorization.k8s.io/v1beta1) > > kind: ClusterRole > > metadata: > > name: filebeat > > labels: > > k8s-app: filebeat > > rules: > > > > - apiGroups: [""] # "" indicates the core API group > > resources: > > - namespaces > > - pods > > verbs: > > - get > > - watch > > - list > > > > * * * > > > > apiVersion: v1 > > kind: ServiceAccount > > metadata: > > name: filebeat > > namespace: kube-system > > labels: > > k8s-app: filebeat ---
**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\ **Post date:** [June 16, 2019, 8:47pm UTC](https://discuss.elastic.co/t/filebeat-installed-as-daemon-set-in-ibm-private-cloud-kubernetes-cluster-dont-send-logs-to-elastic-cloud-instance/184406/4 "2019-06-16T20:47:11Z")
Hi @l.rava, Thank you for your feedback! Our default manifests expect logs under `/var/lib/docker/containers`, as it's the most common scenario, I'm wondering if that's the case for IBM Cloud. Could you please ssh one of the nodes and check how the logs are stored? You can go to `/var/log/pods` and check log files there. They may be common files or symlinks to some other folder. ---
**Author:** ![Ben\_Stucke](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ben_stucke/32/95009_2.png) [@Ben\_Stucke](https://discuss.elastic.co/u/Ben_Stucke)\ **Post date:** [September 24, 2021, 3:30pm UTC](https://discuss.elastic.co/t/filebeat-installed-as-daemon-set-in-ibm-private-cloud-kubernetes-cluster-dont-send-logs-to-elastic-cloud-instance/184406/5 "2021-09-24T15:30:09Z")
@l.rava Have you solved the problem. I have have the same problem. I am using filebeat 7.15.0 and used the guide on [Run Filebeat on Kubernetes | Filebeat Reference [7.15] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/running-on-kubernetes.html). My kubernetes Version on IBM Cloud is 1.19.14\_1557. I have checked that my logs on the kubernetes node are stored on "/var/log/containers". If i go into the filebeat container to the the corresponding path all log files are there too. Heartbeat and Metricbeat works fine and filebeat works with this configuration on minikube. I don't see any errors but my Elasticsearch instance on [elastic.co](http://elastic.co) does not have any logs. ---
**Author:** ![Ben\_Stucke](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ben_stucke/32/95009_2.png) [@Ben\_Stucke](https://discuss.elastic.co/u/Ben_Stucke)\ **Post date:** [October 15, 2021, 1:31pm UTC](https://discuss.elastic.co/t/filebeat-installed-as-daemon-set-in-ibm-private-cloud-kubernetes-cluster-dont-send-logs-to-elastic-cloud-instance/184406/6 "2021-10-15T13:31:20Z")
Setting the volumeMounts and volumes solved it for me: ```auto volumeMounts: - name: config mountPath: /etc/filebeat.yml readOnly: true subPath: filebeat.yml - name: data mountPath: /usr/share/filebeat/data - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true - name: varlog mountPath: /var/log/containers readOnly: true - name: varlogpods mountPath: /var/log/pods readOnly: true volumes: - name: config configMap: defaultMode: 0640 name: filebeat-config - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers - name: varlog hostPath: path: /var/log/containers - name: varlogpods hostPath: path: /var/log/pods # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart - name: data hostPath: # When filebeat runs as non-root user, this directory needs to be writable by group (g+w). path: /var/lib/filebeat-data type: DirectoryOrCreate ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2022, 7:11am UTC](https://discuss.elastic.co/t/filebeat-installed-as-daemon-set-in-ibm-private-cloud-kubernetes-cluster-dont-send-logs-to-elastic-cloud-instance/184406/7 "2022-11-04T07:11:01Z")