# No logs shipped from Kubernetes Filebeat
**URL:** https://discuss.elastic.co/t/no-logs-shipped-from-kubernetes-filebeat/215899
**Category:** Beats
**Tags:** docker, filebeat
**Created:** [January 21, 2020, 2:45pm UTC](https://discuss.elastic.co/t/no-logs-shipped-from-kubernetes-filebeat/215899 "2020-01-21T14:45:05Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@jmgirven](https://discuss.elastic.co/u/jmgirven)
#### Post date: [January 21, 2020, 2:45pm UTC](https://discuss.elastic.co/t/no-logs-shipped-from-kubernetes-filebeat/215899/1 "2020-01-21T14:45:05Z")
I am running filebeats 7.5.1 on my kubernetes cluster as described in the section "Get started in minutes" "Hosted" "Kubernetes Logs" from here:
[https://www.elastic.co/what-is/kubernetes-monitoring](https://www.elastic.co/what-is/kubernetes-monitoring)
With the exception that I am using the `filebeat-kubernetes.yaml` for version 7.5.1 instead of the referenced 6.0 file (which I discovered [here](https://www.elastic.co/guide/en/beats/filebeat/current/running-on-kubernetes.html)).
I am trying to ship my pod logs to my hosted [elastic.co](http://elastic.co) cluster. The filebeat deamonset has all the pods running, and there are no obvious error logs, but I am not seeing any logs in the [elastic.co](http://elastic.co) Kibana logs dashboard. What am I missing?
---
### Author:  [@jsoriano](https://discuss.elastic.co/u/jsoriano)
#### Post date: [January 21, 2020, 4:42pm UTC](https://discuss.elastic.co/t/no-logs-shipped-from-kubernetes-filebeat/215899/2 "2020-01-21T16:42:14Z")
Hi @jmgirven and welcome to discuss! 🙂
> [@jmgirven](#):
>
> With the exception that I am using the `filebeat-kubernetes.yaml` for version 7.5.1 instead of the referenced 6.0 file (which I discovered [here](https://www.elastic.co/guide/en/beats/filebeat/current/running-on-kubernetes.html)).
Oops, sorry! we should update this link.
> [@jmgirven](#):
>
> I am trying to ship my pod logs to my hosted [elastic.co](http://elastic.co) cluster. The filebeat deamonset has all the pods running, and there are no obvious error logs, but I am not seeing any logs in the [elastic.co](http://elastic.co) Kibana logs dashboard. What am I missing?
How is your Kubernetes cluster deployed? Does it have something special like the use of some logging driver?
Could you share the startup logs of one of your filebeat pods?
---
### Author:  [@jmgirven](https://discuss.elastic.co/u/jmgirven)
#### Post date: [January 21, 2020, 5:18pm UTC](https://discuss.elastic.co/t/no-logs-shipped-from-kubernetes-filebeat/215899/3 "2020-01-21T17:18:26Z")
Our Kubernetes cluster is a managed service running on [Oracle Container Engine for Kubernetes](https://www.oracle.com/uk/cloud/compute/container-engine-kubernetes.html). I don't believe it has a special logging driver, but it might. Is there a e.g. `kubectl` command to check?
Sure, does this help:
> 2020-01-21T14:13:04.849Z INFO instance/beat.go:610 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
> 2020-01-21T14:13:04.849Z INFO instance/beat.go:618 Beat ID: 6a60c02e-4e8f-4f65-b236-004b8b1be79e
> 2020-01-21T14:13:04.855Z INFO add\_cloud\_metadata/add\_cloud\_metadata.go:89 add\_cloud\_metadata: hosting provider type not detected.
> 2020-01-21T14:13:04.856Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
> 2020-01-21T14:13:04.856Z INFO [beat] instance/beat.go:941 Beat info {"system\_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "6a60c02e-4e8f-4f65-b236-004b8b1be79e"}}}
> 2020-01-21T14:13:04.856Z INFO [beat] instance/beat.go:950 Build info {"system\_info": {"build": {"commit": "60dd883ca29e1fdd5b8b075bd5f3698948b1d44d", "libbeat": "7.5.1", "time": "2019-12-16T21:56:14.000Z", "version": "7.5.1"}}}
> 2020-01-21T14:13:04.856Z INFO [beat] instance/beat.go:953 Go runtime info {"system\_info": {"go": {"os":"linux","arch":"amd64","max\_procs":4,"version":"go1.12.12"}}}
> 2020-01-21T14:13:04.859Z INFO [beat] instance/beat.go:957 Host info {"system\_info": {"host": {"architecture":"x86\_64","boot\_time":"2020-01-14T10:22:07Z","containerized":false,"name":"oke-c4tinjsgjsd-n2tiyjsgq4d-sj5pd2epukq-1","ip":["127.0.0.1/8","10.0.10.8/24","172.17.0.1/16","10.244.4.0/32","10.244.4.1/24"],"kernel\_version":"4.14.35-1902.8.4.el7uek.x86\_64","mac":["02:00:17:01:e2:ff","02:42:2a:56:83:0f","fe:46:f0:73:f5:df","4e:26:82:f4:b3:e2","7a:7c:03:50:66:da","ae:5e:5c:d7:d4:3e","c2:6a:3e:96:78:d6","42:3b:85:53:a2:c9","6a:ad:09:33:09:6a","56:0a:0c:35:82:3d","76:25:24:e1:bb:24","6e:c3:e8:eb:8b:f1","a2:92:41:1c:7c:fc","7e:3b:e8:a2:1c:70","fe:86:c9:09:66:23","0e:e6:f9:a6:b0:ed","8e:ec:49:a4:e8:62","9a:0a:53:2d:12:70","16:df:85:53:2f:8f","ba:6a:5c:ff:1c:8f","02:38:13:4a:72:3d","0e:a8:50:1a:d1:2e","8e:63:52:2a:c0:a4","56:25:61:93:47:ea"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":7,"patch":1908,"codename":"Core"},"timezone":"UTC","timezone\_offset\_sec":0}}}
> 2020-01-21T14:13:04.859Z INFO [beat] instance/beat.go:986 Process info {"system\_info": {"process": {"capabilities": {"inheritable":["chown","dac\_override","fowner","fsetid","kill","setgid","setuid","setpcap","net\_bind\_service","net\_raw","sys\_chroot","mknod","audit\_write","setfcap"],"permitted":["chown","dac\_override","fowner","fsetid","kill","setgid","setuid","setpcap","net\_bind\_service","net\_raw","sys\_chroot","mknod","audit\_write","setfcap"],"effective":["chown","dac\_override","fowner","fsetid","kill","setgid","setuid","setpcap","net\_bind\_service","net\_raw","sys\_chroot","mknod","audit\_write","setfcap"],"bounding":["chown","dac\_override","fowner","fsetid","kill","setgid","setuid","setpcap","net\_bind\_service","net\_raw","sys\_chroot","mknod","audit\_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no\_new\_privs":true}, "start\_time": "2020-01-21T14:13:04.290Z"}}}
> 2020-01-21T14:13:04.860Z INFO instance/beat.go:297 Setup Beat: filebeat; Version: 7.5.1
> 2020-01-21T14:13:04.860Z INFO [index-management] idxmgmt/std.go:182 Set output.elasticsearch.index to 'filebeat-7.5.1' as ILM is enabled.
> 2020-01-21T14:13:04.860Z INFO elasticsearch/client.go:171 Elasticsearch url: [https://33dd2e19ea264ae3ac3cb95ee80c6d1c.europe-west2.gcp.elastic-cloud.com:443](https://33dd2e19ea264ae3ac3cb95ee80c6d1c.europe-west2.gcp.elastic-cloud.com:443)
> 2020-01-21T14:13:04.860Z INFO [publisher] pipeline/module.go:97 Beat name: oke-c4tinjsgjsd-n2tiyjsgq4d-sj5pd2epukq-1
> 2020-01-21T14:13:04.861Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
> 2020-01-21T14:13:04.861Z INFO instance/beat.go:429 filebeat start running.
> 2020-01-21T14:13:04.862Z INFO registrar/registrar.go:145 Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
> 2020-01-21T14:13:04.862Z INFO registrar/registrar.go:152 States Loaded from registrar: 0
> 2020-01-21T14:13:04.862Z INFO crawler/crawler.go:72 Loading Inputs: 1
> 2020-01-21T14:13:04.873Z INFO add\_kubernetes\_metadata/kubernetes.go:68 add\_kubernetes\_metadata: kubernetes env detected, with version: v1.14.8
> 2020-01-21T14:13:04.873Z INFO kubernetes/util.go:79 kubernetes: Using node 10.0.10.8 provided in the config
> 2020-01-21T14:13:04.974Z INFO log/input.go:152 Configured paths: [/var/log/containers/\*.log]
> 2020-01-21T14:13:04.974Z INFO input/input.go:114 Starting input of type: container; ID: 3960813757973143954
> 2020-01-21T14:13:04.974Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
> 2020-01-21T14:13:34.864Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":40,"time":{"ms":50}},"total":{"ticks":160,"time":{"ms":171},"value":160},"user":{"ticks":120,"time":{"ms":121}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral\_id":"c2be0d79-bb4e-4544-9d94-a72d6dbfd4b5","uptime":{"ms":30041}},"memstats":{"gc\_next":10911632,"memory\_alloc":5629496,"memory\_total":16939904,"rss":55189504},"runtime":{"goroutines":33}},"filebeat":{"harvester":{"open\_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":4},"load":{"1":0.06,"15":0.31,"5":0.17,"norm":{"1":0.015,"15":0.0775,"5":0.0425}}}}}}
> 2020-01-21T14:14:04.864Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":50,"time":{"ms":9}},"total":{"ticks":180,"time":{"ms":22},"value":180},"user":{"ticks":130,"time":{"ms":13}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral\_id":"c2be0d79-bb4e-4544-9d94-a72d6dbfd4b5","uptime":{"ms":60041}},"memstats":{"gc\_next":10911632,"memory\_alloc":6297736,"memory\_total":17608144},"runtime":{"goroutines":33}},"filebeat":{"harvester":{"open\_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.17,"15":0.31,"5":0.18,"norm":{"1":0.0425,"15":0.0775,"5":0.045}}}}}}
> 2020-01-21T14:14:34.864Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":60,"time":{"ms":11}},"total":{"ticks":200,"time":{"ms":23},"value":200},"user":{"ticks":140,"time":{"ms":12}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral\_id":"c2be0d79-bb4e-4544-9d94-a72d6dbfd4b5","uptime":{"ms":90041}},"memstats":{"gc\_next":10911632,"memory\_alloc":6760848,"memory\_total":18071256},"runtime":{"goroutines":33}},"filebeat":{"harvester":{"open\_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.1,"15":0.3,"5":0.17,"norm":{"1":0.025,"15":0.075,"5":0.0425}}}}}}
> 2020-01-21T14:15:04.864Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":80,"time":{"ms":10}},"total":{"ticks":230,"time":{"ms":22},"value":230},"user":{"ticks":150,"time":{"ms":12}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral\_id":"c2be0d79-bb4e-4544-9d94-a72d6dbfd4b5","uptime":{"ms":120041}},"memstats":{"gc\_next":10911632,"memory\_alloc":7309504,"memory\_total":18619912},"runtime":{"goroutines":33}},"filebeat":{"harvester":{"open\_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.14,"15":0.29,"5":0.16,"norm":{"1":0.035,"15":0.0725,"5":0.04}}}}}}
@jsoriano
---
### Author:  [@jsoriano](https://discuss.elastic.co/u/jsoriano)
#### Post date: [January 21, 2020, 6:22pm UTC](https://discuss.elastic.co/t/no-logs-shipped-from-kubernetes-filebeat/215899/4 "2020-01-21T18:22:19Z")
Could you check if the docker logs in your machine are under `/var/log/containers`?
If you cannot find your logs you can try to login into one of your kubernetes nodes and use `docker inspect` to check the configuration and the log path of one of your running containers.
Something else you can also try is to use Beats [autodiscover](https://www.elastic.co/guide/en/beats/filebeat/7.5/configuration-autodiscover.html), for that follow the instructions in the manifest you downloaded.
---
### Author:  [@jmgirven](https://discuss.elastic.co/u/jmgirven)
#### Post date: [January 22, 2020, 10:16am UTC](https://discuss.elastic.co/t/no-logs-shipped-from-kubernetes-filebeat/215899/5 "2020-01-22T10:16:14Z")
If I list the `/var/log/containers` directory mounted into one of the filebeat containers, e.g:
```
$ kubectl -n kube-system exec -it filebeat-xyzab -- ls -l /var/log/containers/
```
I can see the many log files for my containers there. The files are all symlinks to e.g: `/var/log/pods/namespace_podname-xyzcodezyx/podname/0.log`, which seems fine because `/var/log` is mounted as a volume into the filebeat container. However this is also a symlink in my case, to e.g: `/u01/data/docker/containers/xyzxyz/zyxzyx-json.log`, which is not a filebeat volume.
I have copied the volume mount configuration for `/var/lib/docker/containers` except replaced it with `/u01/data/docker/containers`, and recreated my filebeat daemonset. Now the logs are coming through to [elastic.co](http://elastic.co), thanks. I assume this is just some specific configuration to OKE.
---
### Author:  [@jsoriano](https://discuss.elastic.co/u/jsoriano)
#### Post date: [January 22, 2020, 11:48am UTC](https://discuss.elastic.co/t/no-logs-shipped-from-kubernetes-filebeat/215899/6 "2020-01-22T11:48:02Z")
Great, thanks for posting the solution you have found!
---
### Author:  [@Vinicius\_Maia](https://discuss.elastic.co/u/Vinicius_Maia)
#### Post date: [February 10, 2020, 7:40pm UTC](https://discuss.elastic.co/t/no-logs-shipped-from-kubernetes-filebeat/215899/7 "2020-02-10T19:40:07Z")
> [@jmgirven](#):
>
> /var/log/pods
Hi jmgirven, I replaced the config that you comment (/u01/data/docker/containers) but I didn't have success.
I'm using OKE as you and my elasticsearch don't receive filebeat index.
Following, my filebeat.yaml
```
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: elk
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.inputs:
- type: container
paths:
- /u01/data/docker/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/u01/data/docker/containers/"
# To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
#filebeat.autodiscover:
# providers:
# - type: kubernetes
# host: ${NODE_NAME}
# hints.enabled: true
# hints.default_config:
# type: container
# paths:
# - /u01/data/docker/containers/*${data.kubernetes.container.id}.log
processors:
- add_cloud_metadata:
- add_host_metadata:
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:MY_IP_ADDRESS}:${ELASTICSEARCH_PORT:9200}']
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: filebeat
namespace: elk
labels:
k8s-app: filebeat
spec:
selector:
matchLabels:
k8s-app: filebeat
template:
metadata:
labels:
k8s-app: filebeat
spec:
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat:7.5.2
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
env:
- name: ELASTICSEARCH_HOST
value: 10.2.0.22
- name: ELASTICSEARCH_PORT
value: "9200"
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
runAsUser: 0
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
- name: data
mountPath: /usr/share/filebeat/data
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: varlog
mountPath: /var/log
readOnly: true
volumes:
- name: config
configMap:
defaultMode: 0600
name: filebeat-config
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: varlog
hostPath:
path: /var/log
# data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
- name: data
hostPath:
path: /var/lib/filebeat-data
type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:
- kind: ServiceAccount
name: filebeat
namespace: elk
roleRef:
kind: ClusterRole
name: filebeat
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: filebeat
labels:
k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- namespaces
- pods
verbs:
- get
- watch
- list
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: elk
labels:
k8s-app: filebeat
---
```
---
### Author:  [@Vinicius\_Maia](https://discuss.elastic.co/u/Vinicius_Maia)
#### Post date: [February 11, 2020, 1:34pm UTC](https://discuss.elastic.co/t/no-logs-shipped-from-kubernetes-filebeat/215899/8 "2020-02-11T13:34:25Z")
Finally I fixed my setup and have filebeat sending data to Elasticsearch.
I had mount in my OKE, the path: /u01/data/docker in the volumeMounts and volumes in the name: varlibdockercontainers variables.
Thanks to @jmgirven for the tip.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 10, 2020, 1:34pm UTC](https://discuss.elastic.co/t/no-logs-shipped-from-kubernetes-filebeat/215899/9 "2020-03-10T13:34:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.