# Filebeat stoped to write logs after K8S upgrade to 1.14.1 ver
**URL:** https://discuss.elastic.co/t/filebeat-stoped-to-write-logs-after-k8s-upgrade-to-1-14-1-ver/183870
**Category:** Beats
**Tags:** filebeat
**Created:** [June 2, 2019, 8:34pm UTC](https://discuss.elastic.co/t/filebeat-stoped-to-write-logs-after-k8s-upgrade-to-1-14-1-ver/183870 "2019-06-02T20:34:14Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@aleksei.saiko](https://discuss.elastic.co/u/aleksei.saiko)
#### Post date: [June 2, 2019, 8:34pm UTC](https://discuss.elastic.co/t/filebeat-stoped-to-write-logs-after-k8s-upgrade-to-1-14-1-ver/183870/1 "2019-06-02T20:34:14Z")
Hi there!
After upgrading the K8S ver from 1.10 to 1.14.1 ver, our filebeat stoped to write logs.
I did found something in k8s doc - The container log directory changed from `/var/lib/docker/` to `/var/log/pods/` . If you use your own logging solution that monitors the previous directory, update accordingly.
But still, not working.
/var/log/pods contatins "containers" folder, which is empty.
Could you please advice? (ELK ver 6.6.2)
Attaching my DS yaml
```auto
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
creationTimestamp: "2018-06-26T10:40:05Z"
generation: 11
labels:
k8s-app: filebeat
kubernetes.io/cluster-service: "true"
name: filebeat
namespace: kube-system
resourceVersion: "67464142"
selfLink: /apis/extensions/v1beta1/namespaces/kube-system/daemonsets/filebeat
uid: 4a0a1e57-792d-11e8-9321-528d05d8e1e1
spec:
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: filebeat
kubernetes.io/cluster-service: "true"
template:
metadata:
creationTimestamp: null
labels:
k8s-app: filebeat
kubernetes.io/cluster-service: "true"
spec:
containers:
- args:
- -c
- /etc/filebeat.yml
- -e
env:
- name: LOGSTASH_HOSTS
value: logstash-kube:5044
image: docker.elastic.co/beats/filebeat:6.6.2
imagePullPolicy: IfNotPresent
name: filebeat
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
securityContext:
procMount: Default
runAsUser: 0
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/filebeat.yml
name: config
readOnly: true
subPath: filebeat.yml
- mountPath: /usr/share/filebeat/prospectors.d
name: prospectors
readOnly: true
- mountPath: /usr/share/filebeat/data
name: data
- mountPath: /var/log/pods/containers
name: varlibdockercontainers
readOnly: true
- mountPath: /var/log
name: varlog
readOnly: true
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: filebeat
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
volumes:
- configMap:
defaultMode: 384
name: filebeat-config
name: config
- hostPath:
path: /var/log
type: ""
name: varlog
- hostPath:
path: /var/log/pods/containers
type: ""
name: varlibdockercontainers
- configMap:
defaultMode: 384
name: filebeat-prospectors
name: prospectors
- emptyDir: {}
name: data
templateGeneration: 11
updateStrategy:
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate
status:
currentNumberScheduled: 6
desiredNumberScheduled: 6
numberAvailable: 6
numberMisscheduled: 0
numberReady: 6
observedGeneration: 10
updatedNumberScheduled: 6
```
Configs
```auto
apiVersion: v1
data:
kubernetes.yml: |-
- type: docker
containers.ids:
- "*"
processors:
- add_kubernetes_metadata:
in_cluster: true
scan_frequency: 10s
close_inactive: 1m
kind: ConfigMap
metadata:
creationTimestamp: "2019-02-19T19:55:05Z"
labels:
k8s-app: filebeat
kubernetes.io/cluster-service: "true"
name: filebeat-prospectors
namespace: kube-system
resourceVersion: "42416935"
selfLink: /api/v1/namespaces/kube-system/configmaps/filebeat-prospectors
uid: 40568db5-3480-11e9-9194-72f87630c359
```
And
```auto
apiVersion: v1
data:
filebeat.yml: |-
filebeat.config:
prospectors:
# Mounted `filebeat-prospectors` configmap:
path: ${path.config}/prospectors.d/*.yml
# Reload prospectors configs as they change:
reload.enabled: false
modules:
path: ${path.config}/modules.d/*.yml
# Reload module configs as they change:
reload.enabled: false
output.logstash:
hosts: ${LOGSTASH_HOSTS:?No logstash host configured. Use env var LOGSTASH_HOSTS to set hosts.}
kind: ConfigMap
metadata:
creationTimestamp: "2018-06-26T10:39:24Z"
labels:
k8s-app: filebeat
kubernetes.io/cluster-service: "true"
name: filebeat-config
namespace: kube-system
resourceVersion: "130661"
selfLink: /api/v1/namespaces/kube-system/configmaps/filebeat-config
```
Could you advice please? Thx!
Aleksei
---
### Author:  [@aleksei.saiko](https://discuss.elastic.co/u/aleksei.saiko)
#### Post date: [June 4, 2019, 6:14am UTC](https://discuss.elastic.co/t/filebeat-stoped-to-write-logs-after-k8s-upgrade-to-1-14-1-ver/183870/2 "2019-06-04T06:14:19Z")
Anyone? Someone experienced the same problem?
---
### Author:  [@exekias](https://discuss.elastic.co/u/exekias)
#### Post date: [June 4, 2019, 7:49am UTC](https://discuss.elastic.co/t/filebeat-stoped-to-write-logs-after-k8s-upgrade-to-1-14-1-ver/183870/3 "2019-06-04T07:49:27Z")
Hi @aleksei.saiko,
We are introducing a new input to support this use case, more details here: [https://github.com/elastic/beats/pull/12162](https://github.com/elastic/beats/pull/12162)
It should be out with 7.2 in the following weeks. Something you could use in the meantime:
```auto
- type: log
paths:
- /var/log/containers/*.log
symlinks: true
json.message_key: log
json.keys_under_root: true
```
---
### Author:  [@aleksei.saiko](https://discuss.elastic.co/u/aleksei.saiko)
#### Post date: [June 4, 2019, 10:00am UTC](https://discuss.elastic.co/t/filebeat-stoped-to-write-logs-after-k8s-upgrade-to-1-14-1-ver/183870/4 "2019-06-04T10:00:46Z")
Thanks for your answer!
I guess I will need to put this path in Daemon yaml as well in
```auto
volumeMounts:
- mountPath: ...
```
?
---
### Author:  [@exekias](https://discuss.elastic.co/u/exekias)
#### Post date: [June 7, 2019, 8:11am UTC](https://discuss.elastic.co/t/filebeat-stoped-to-write-logs-after-k8s-upgrade-to-1-14-1-ver/183870/5 "2019-06-07T08:11:15Z")
yes!, also check if they are symlinks, where they are pointing to, probably `/var/lib/docker/containers`. Then you will need to have both mounted
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2019, 8:11am UTC](https://discuss.elastic.co/t/filebeat-stoped-to-write-logs-after-k8s-upgrade-to-1-14-1-ver/183870/6 "2019-07-05T08:11:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.