Filebeat installed as Daemon Set in IBM Private Cloud Kubernetes Cluster don't send logs to Elastic Cloud instance

Hi
I installed as Daemon Set metricBeat and fileBeat into my infrastracture:

  • IBM Private Cloud
  • kubernetes cluster - 2 nodes - v.1.13
    following this guide.

MetricBeat send data without problem. Filebeat don't send data.

here logs in debug for filebeat:

INFO instance/beat.go:571 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
DEBUG [beat] instance/beat.go:623 Beat metadata path: /usr/share/filebeat/data/meta.json
INFO instance/beat.go:579 Beat ID: 8498bd87-16c3-42a3-b5e0-4ca23133a429
INFO [index-management.ilm] ilm/ilm.go:129 Policy name: filebeat-7.1.1
DEBUG [filters] add_cloud_metadata/add_cloud_metadata.go:164 add_cloud_metadata: starting to fetch metadata, timeout=3s
DEBUG [filters] add_cloud_metadata/add_cloud_metadata.go:196 add_cloud_metadata: received disposition for qcloud after 339.455729ms. result=[provider:qcloud, error=failed requesting qcloud metadata: Get http://metadata.tencentyun.com/meta-data/instance-id: dial tcp: lookup metadata.tencentyun.com on 172.21.0.10:53: no such host, metadata={}]
DEBUG [filters] add_cloud_metadata/add_cloud_metadata.go:196 add_cloud_metadata: received disposition for openstack after 3.000222149s. result=[provider:openstack, error=failed requesting openstack metadata: Get http://169.254.169.254/2009-04-04/meta-data/hostname: dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
DEBUG [filters] add_cloud_metadata/add_cloud_metadata.go:203 add_cloud_metadata: timed-out waiting for all responses
DEBUG [filters] add_cloud_metadata/add_cloud_metadata.go:167 add_cloud_metadata: fetchMetadata ran for 3.000347097s
INFO add_cloud_metadata/add_cloud_metadata.go:346 add_cloud_metadata: hosting provider type not detected.
DEBUG [processors] processors/processor.go:66 Processors: add_cloud_metadata=null
DEBUG [seccomp] seccomp/seccomp.go:109 Loading syscall filter {"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4",...,"writev"],"action":"allow"}]}}}
INFO [seccomp] seccomp/seccomp.go:116 Syscall filter successfully installed
INFO [beat] instance/beat.go:827 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "8498bd87-16c3-42a3-b5e0-4ca23133a429"}}}
INFO [beat] instance/beat.go:836 Build info {"system_info": {"build": {"commit": "3358d9a5a09e3c6709a2d3aaafde628ea34e8419", "libbeat": "7.1.1", "time": "2019-05-23T13:21:33.000Z", "version": "7.1.1"}}}
INFO [beat] instance/beat.go:839 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.11.5"}}}
INFO [beat] instance/beat.go:843 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-04-26T14:06:48Z","containerized":false,"name":"kube-fra02-cr7a*************49fc787c-w5.cloud.ibm","ip":["127.0.0.1/8","172.20.0.1/32","::1/128","10.XX.225.43/26","fe80::467:43ff:fec8:f89c/64","158.XX.138.101/28","158.XX.145.222/32","fe80::xxx:72ff:fe53:6022/64","127.0.0.10/31","fe80::bcaa:c3ff:fe81:c38/64","172.30.7.64/32"}}}
INFO [beat] instance/beat.go:872 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter"}, "start_time": "2019-06-05T12:35:27.460Z"}}}
INFO instance/beat.go:280 Setup Beat: filebeat; Version: 7.1.1
DEBUG [beat] instance/beat.go:301 Initializing output plugins
INFO [index-management] idxmgmt/std.go:165 Set output.elasticsearch.index to 'filebeat-7.1.1' as ILM is enabled.
INFO elasticsearch/client.go:165 Elasticsearch url: https://51fxxxxxxxxxxxxf3c92201363c.europe-west1.gcp.cloud.es.io:443
DEBUG [publisher] pipeline/consumer.go:137 start pipeline event consumer
INFO [publisher] pipeline/module.go:97 Beat name: kube-fra02-cr7a4aeac0xxxxxxx40c8c9799d049fc787c-w5.cloud.ibm
INFO instance/beat.go:391 filebeat start running.
DEBUG [test] registrar/migrate.go:159 isFile(/usr/share/filebeat/data/registry) -> false
DEBUG [test] registrar/migrate.go:159 isFile() -> false
DEBUG [test] registrar/migrate.go:152 isDir(/usr/share/filebeat/data/registry/filebeat) -> true
DEBUG [test] registrar/migrate.go:159 isFile(/usr/share/filebeat/data/registry/filebeat/meta.json) -> true
INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
DEBUG [registrar] registrar/migrate.go:51 Registry type '0' found
DEBUG [registrar] registrar/registrar.go:125 Registry file set to: /usr/share/filebeat/data/registry/filebeat/data.json
INFO registrar/registrar.go:145 Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
INFO registrar/registrar.go:152 States Loaded from registrar: 0
INFO crawler/crawler.go:72 Loading Inputs: 0
INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 0
WARN [cfgwarn] kubernetes/kubernetes.go:55 BETA: The kubernetes autodiscover is beta
INFO kubernetes/util.go:86 kubernetes: Using pod name kube-fra02-cr7a4aeacxxxxx799d049fc787c-w5.cloud.ibm and namespace kube-system to discover kubernetes node
INFO cfgfile/reload.go:150 Config reloader started
DEBUG [registrar] registrar/registrar.go:278 Starting Registrar
ERROR kubernetes/util.go:90 kubernetes: Querying for pod failed with error: kubernetes api: Failure 404 pods "kube-fra02-cr7a4aeac036c7440c8c9799d049fc787c-w5.cloud.ibm" not found

deploy yaml content:

Hi @l.rava,

Thank you for your feedback! Our default manifests expect logs under /var/lib/docker/containers, as it's the most common scenario, I'm wondering if that's the case for IBM Cloud. Could you please ssh one of the nodes and check how the logs are stored?

You can go to /var/log/pods and check log files there. They may be common files or symlinks to some other folder.