# JournalBeat Stopped Shipping Logs **URL:** \ **Category:** Beats\ **Tags:** journalbeat\ **Created:** [December 20, 2020, 1:38pm UTC](https://discuss.elastic.co/t/journalbeat-stopped-shipping-logs/259195 "2020-12-20T13:38:15Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![vamshisiddarth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vamshisiddarth/32/82759_2.png) [@vamshisiddarth](https://discuss.elastic.co/u/vamshisiddarth)\ **Post date:** [December 20, 2020, 1:38pm UTC](https://discuss.elastic.co/t/journalbeat-stopped-shipping-logs/259195/1 "2020-12-20T13:38:15Z")
JournalBeat stopped shipping logs on all environments for us from 19th Dec, 2020. We configured our beats services to ship logs from aws instance to elasticsearch through logstash. Our filebeat and metricbeat logs are being shipped fine. It has issue only shipping the journalbeat logs. We have verified the SELinuxOptions as well. We tried enabling the debug logs on elasticsearch, logstash and journalbeat as well. Not finding any errors but still no logs in the elasticsearch. ELK Stack version: 7.10.1 JournalBeats version: 7.10.1 Kubernetes version: 1.19.4 Below is our kubernetes manifest for the journalbeat. ```auto --- apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default name: journalbeat namespace: kube-system spec: allowedCapabilities: - KILL - CHOWN - FSETID - FOWNER - SETGID - SETUID - SETFCAP - SETPCAP - AUDIT_WRITE - NET_BIND_SERVICE fsGroup: rule: RunAsAny hostIPC: false hostNetwork: false hostPID: false privileged: false requiredDropCapabilities: - MKNOD - DAC_OVERRIDE - NET_RAW - SYS_CHROOT runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny volumes: - secret - configMap - hostPath --- apiVersion: v1 kind: ServiceAccount metadata: name: journalbeat namespace: kube-system --- apiVersion: v1 kind: ConfigMap metadata: name: journalbeat-config namespace: kube-system labels: k8s-app: journalbeat-logging version: v1 data: journalbeat.yml: | name: "${NODENAME}" journalbeat.inputs: - paths: [] seek: cursor cursor_seek_fallback: tail processors: - add_kubernetes_metadata: host: "${NODENAME}" in_cluster: true default_indexers.enabled: false default_matchers.enabled: false indexers: - container: matchers: - fields: lookup_fields: ["container.id"] - decode_json_fields: fields: ["message"] process_array: false max_depth: 1 target: "" overwrite_keys: true - drop_event.when: or: - regexp.kubernetes.pod.name: "filebeat-.*" - regexp.kubernetes.pod.name: "journalbeat-.*" - regexp.kubernetes.pod.name: "metrics-server-.*" - equals.syslog.identifier: "audit" - regexp.message: '^.*?\baudit\b.*?$' - and: - equals.systemd.unit: "docker.service" - or: - regexp.message: '^.*?\bJournalbeat is experimental.*?$' - and: - equals.systemd.unit: "init.scope" - or: - regexp.message: '^.*?\bStarted libcontainer container.*?$' - equals.message: "Started rpm-ostree System Management Daemon." - regexp.message: '^.*?\bSucceeded.*?$' # Updated to ignore conflict of indices between environments setup.ilm.enabled: false setup.template.enabled: false output.logstash: hosts: '${LOGSTASH_HOSTS}' compression_level: 7 --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: journalbeat rules: - apiGroups: - "" resources: - namespaces - pods verbs: - get - watch - list - apiGroups: - extensions resourceNames: - journalbeat resources: - podsecuritypolicies verbs: - use --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: journalbeat roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: journalbeat subjects: - kind: ServiceAccount name: journalbeat namespace: kube-system --- apiVersion: apps/v1 kind: DaemonSet metadata: name: journalbeat namespace: kube-system labels: k8s-app: journalbeat-logging version: v1 spec: selector: matchLabels: k8s-app: journalbeat-logging version: v1 template: metadata: labels: k8s-app: journalbeat-logging version: v1 app: journalbeat name: journalbeat spec: containers: - args: - -e - -c - /etc/journalbeat.yml command: - journalbeat env: - name: NODENAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: PODNAME valueFrom: fieldRef: fieldPath: metadata.name - name: LOGSTASH_HOSTS value: logstash.company.com - name: LOGSTASH_PORT value: "5044" - name: ENVIRONMENT value: prod image: docker.elastic.co/beats/journalbeat:7.10.1 imagePullPolicy: Always name: journalbeat resources: limits: cpu: 600m memory: 800Mi requests: cpu: 200m memory: 400Mi volumeMounts: - mountPath: /usr/share/journalbeat/data name: data - mountPath: /var/log/journal name: var-journal - mountPath: /run/log/journal name: run-journal - mountPath: /etc/journalbeat.yml name: config subPath: journalbeat.yml - mountPath: /etc/machine-id name: machine-id hostNetwork: true nodeSelector: {} securityContext: seLinuxOptions: user: system_u role: system_r type: spc_t level: s0 fsGroup: 0 runAsUser: 0 serviceAccountName: journalbeat terminationGracePeriodSeconds: 60 tolerations: - effect: NoSchedule key: node-role.kubernetes.io/master volumes: - hostPath: path: /var/log/journal/journalbeat-data name: data - hostPath: path: /var/log/journal name: var-journal - hostPath: path: /run/log/journal name: run-journal - hostPath: path: /etc/machine-id name: machine-id - configMap: items: - key: journalbeat.yml path: journalbeat.yml name: journalbeat-config name: config ``` ---
**Author:** ![codeaprendiz](https://avatars.discourse-cdn.com/v4/letter/c/7ab992/32.png) [@codeaprendiz](https://discuss.elastic.co/u/codeaprendiz)\ **Post date:** [December 20, 2020, 8:05pm UTC](https://discuss.elastic.co/t/journalbeat-stopped-shipping-logs/259195/2 "2020-12-20T20:05:27Z")
We are facing the same issue. The following are the logs in _DEBUG_ mode. ## DEBUG OUTPUT ```bash $ kubectl logs -f journalbeat-9852r -n kube-system 2020-12-20T19:54:08.609Z INFO instance/beat.go:645 Home path: [/usr/share/journalbeat] Config path: [/usr/share/journalbeat] Data path: [/usr/share/journalbeat/data] Logs path: [/usr/share/journalbeat/logs] 2020-12-20T19:54:08.609Z DEBUG [beat] instance/beat.go:697 Beat metadata path: /usr/share/journalbeat/data/meta.json 2020-12-20T19:54:08.609Z INFO instance/beat.go:653 Beat ID: 905e574e-0920-4b40-ab17-055d1bc2a1ba 2020-12-20T19:54:08.610Z DEBUG [conditions] conditions/conditions.go:98 New condition regexp: map[] 2020-12-20T19:54:08.611Z DEBUG [conditions] conditions/conditions.go:98 New condition equals: map[message:{0 Started rpm-ostree System Management Daemon. false}] 2020-12-20T19:54:08.611Z DEBUG [conditions] conditions/conditions.go:98 New condition regexp: map[] 2020-12-20T19:54:08.612Z DEBUG [conditions] conditions/conditions.go:98 New condition equals: map[systemd.unit:{0 init.scope false}] and regexp: map[] or equals: map[message:{0 Started rpm-ostree System Management Daemon. false}] or regexp: map[] 2020-12-20T19:54:08.612Z DEBUG [conditions] conditions/conditions.go:98 New condition regexp: map[] or regexp: map[] or regexp: map[] or equals: map[syslog.identifier:{0 audit false}] or regexp: map[] or equals: map[systemd.unit:{0 docker.service false}] and regexp: map[] or equals: map[systemd.unit:{0 init.scope false}] and regexp: map[] or equals: map[message:{0 Started rpm-ostree System Management Daemon. false}] or regexp: map[] 2020-12-20T19:54:08.612Z DEBUG [processors] processors/processor.go:120 Generated new processors: add_kubernetes_metadata, decode_json_fields=message, drop_event, condition=regexp: map[] or regexp: map[] or regexp: map[] or equals: map[syslog.identifier:{0 audit false}] or regexp: map[] or equals: map[systemd.unit:{0 docker.service false}] and regexp: map[] or equals: map[systemd.unit:{0 init.scope false}] and regexp: map[] or equals: map[message:{0 Started rpm-ostree System Management Daemon. false}] or regexp: map[] 2020-12-20T19:54:08.612Z DEBUG [seccomp] seccomp/seccomp.go:117 Loading syscall filter {"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","chown","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchmodat","fchown","fchownat","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}} 2020-12-20T19:54:08.612Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed 2020-12-20T19:54:08.612Z INFO [beat] instance/beat.go:981 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/journalbeat", "data": "/usr/share/journalbeat/data", "home": "/usr/share/journalbeat", "logs": "/usr/share/journalbeat/logs"}, "type": "journalbeat", "uuid": "905e574e-0920-4b40-ab17-055d1bc2a1ba"}}} 2020-12-20T19:54:08.612Z INFO [beat] instance/beat.go:993 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.14.7"}}} 2020-12-20T19:54:08.614Z INFO [beat] instance/beat.go:997 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-12-17T20:04:23Z","containerized":true,"name":"ip-10-134-109-2","ip":["127.0.0.1/8","::1/128","10.134.109.2/18","2a05:d018:59e:5801:5e33:4db8:622e:4e6a/128","fe80::14:b9ff:fe99:d8ef/64","172.17.0.1/16","10.136.24.142/32","10.136.175.78/32","10.136.241.110/32","10.136.83.61/32","10.136.52.87/32","10.136.105.175/32","10.136.135.51/32","10.136.27.74/32","10.136.184.145/32","10.136.220.231/32","10.136.80.253/32","10.136.26.147/32","10.136.235.176/32","10.136.93.201/32","10.136.35.104/32","10.136.26.200/32","10.136.46.245/32","10.136.156.12/32","10.136.0.10/32","10.136.198.82/32","10.136.51.141/32","10.136.48.143/32","10.136.234.155/32","10.136.195.0/32","10.136.37.104/32","10.136.143.231/32","10.136.210.147/32","10.136.108.226/32","10.136.91.20/32","10.136.0.1/32","10.136.8.225/32","10.136.148.246/32","10.136.102.34/32","10.136.127.140/32","10.136.220.125/32","10.136.237.12/32","10.2.26.0/32","fe80::90e5:52ff:fec7:eed9/64","10.2.26.1/24","fe80::2021:c4ff:fef6:b1d1/64","fe80::1405:d6ff:fedd:b417/64","fe80::1c1e:2ff:fe63:3f4b/64","fe80::dc01:5ff:fef4:f2d3/64","fe80::2035:2dff:fe1f:f79e/64","fe80::7036:3eff:feaa:7203/64","fe80::c0b1:96ff:fed9:338f/64","fe80::84f7:d5ff:febe:58c2/64","fe80::8493:6cff:fe11:28b9/64"],"kernel_version":"5.9.11-200.fc33.x86_64","mac":["02:14:b9:99:d8:ef","02:42:d6:af:81:eb","9a:8f:68:58:ef:1b","92:e5:52:c7:ee:d9","22:21:c4:f6:b1:d1","fa:91:26:19:8d:f6","56:f8:e8:45:61:64","ce:92:e7:07:c2:50","ca:a8:8c:bc:85:5e","1a:f9:45:c3:3c:1e","56:c8:1d:a3:81:a2","72:9e:6c:a2:ee:7c","ae:30:e2:f7:39:62"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"ec2756fb725f5dcf7bbf13bc22225dd8"}}} 2020-12-20T19:54:08.614Z INFO [beat] instance/beat.go:1026 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/journalbeat", "exe": "/usr/share/journalbeat/journalbeat", "name": "journalbeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-12-20T19:54:08.510Z"}}} 2020-12-20T19:54:08.615Z INFO instance/beat.go:299 Setup Beat: journalbeat; Version: 7.10.0 2020-12-20T19:54:08.615Z DEBUG [beat] instance/beat.go:325 Initializing output plugins 2020-12-20T19:54:08.615Z DEBUG [publisher] pipeline/consumer.go:148 start pipeline event consumer 2020-12-20T19:54:08.615Z INFO [publisher] pipeline/module.go:113 Beat name: ip-10-134-109-2 2020-12-20T19:54:08.615Z WARN [cfgwarn] beater/journalbeat.go:55 EXPERIMENTAL: Journalbeat is experimental. 2020-12-20T19:54:08.616Z INFO add_kubernetes_metadata/kubernetes.go:71 add_kubernetes_metadata: kubernetes env detected, with version: v1.19.3 2020-12-20T19:54:08.616Z INFO [kubernetes] kubernetes/util.go:99 kubernetes: Using node ip-10-134-109-2 provided in the config {"libbeat.processor": "add_kubernetes_metadata"} 2020-12-20T19:54:08.616Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:162 Initializing a new Kubernetes watcher using host: ip-10-134-109-2 {"libbeat.processor": "add_kubernetes_metadata"} 2020-12-20T19:54:08.633Z DEBUG [input] reader/journal.go:75 New journal is opened for reading {"path": "LOCAL_SYSTEM_JOURNAL"} 2020-12-20T19:54:08.633Z DEBUG [input] input/input.go:112 New input is created for paths [] 2020-12-20T19:54:08.633Z INFO instance/beat.go:455 journalbeat start running. 2020-12-20T19:54:08.633Z INFO [journalbeat] beater/journalbeat.go:93 journalbeat is running! Hit CTRL-C to stop it. 2020-12-20T19:54:08.638Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s 2020-12-20T19:54:08.729Z DEBUG [kubernetes] kubernetes/watcher.go:165 cache sync done 2020-12-20T19:54:08.729Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:182 Adding kubernetes pod: frontend/application-www-spot-pod-6b9dcd499b-shg4t {"libbeat.processor": "add_kubernetes_metadata"} 2020-12-20T19:54:08.729Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:258 Created index e8736a2dde3b9310b20f6feada00848ae72e6b4aa27fdf4fd34bc2e7dcf8a30d for pod frontend/application-www-spot-pod-6b9dcd499b-shg4t {"libbeat.processor": "add_kubernetes_metadata"} 2020-12-20T19:54:08.729Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:182 Adding kubernetes pod: frontend/application-backoffice-spot-pod-799d4c7556-6cs7b {"libbeat.processor": "add_kubernetes_metadata"} 2020-12-20T19:54:08.729Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:258 Created index 97372684e4b697c6040a2abcf4dfefbda2e15c3022d0c0a0719f3e6fa7de7c9f for pod frontend/application-backoffice-spot-pod-799d4c7556-6cs7b {"libbeat.processor": "add_kubernetes_metadata"} 2020-12-20T19:54:08.818Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:258 Created index cedef08e8b58bdbb172ec223745e038b574a5f744d6ff6d1765feb9d1d50624b for pod kube-system/journalbeat-9852r {"libbeat.processor": "add_kubernetes_metadata"} 2020-12-20T19:54:38.640Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"id":"docker-cedef08e8b58bdbb172ec223745e038b574a5f744d6ff6d1765feb9d1d50624b.scope"},"cpuacct":{"id":"docker-cedef08e8b58bdbb172ec223745e038b574a5f744d6ff6d1765feb9d1d50624b.scope"},"memory":{"id":"docker-cedef08e8b58bdbb172ec223745e038b574a5f744d6ff6d1765feb9d1d50624b.scope"}},"cpu":{"system":{"ticks":30,"time":{"ms":32}},"total":{"ticks":80,"time":{"ms":88},"value":80},"user":{"ticks":50,"time":{"ms":56}}},"handles":{"limit":{"hard":1024,"soft":1024},"open":39},"info":{"ephemeral_id":"c6282f20-241d-4e69-9762-0633ccb51c4a","uptime":{"ms":30047}},"memstats":{"gc_next":11246624,"memory_alloc":6747480,"memory_total":15841008,"rss":73404416},"runtime":{"goroutines":27}},"journalbeat":{"journals":{"journal_0":{"path":"LOCAL_SYSTEM_JOURNAL","size_in_bytes":3816935424}}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":0}}},"system":{"cpu":{"cores":2},"load":{"1":0.01,"15":0.08,"5":0.08,"norm":{"1":0.005,"15":0.04,"5":0.04}}}}}} 2020-12-20T19:55:08.640Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":30,"time":{"ms":5}},"total":{"ticks":90,"time":{"ms":10},"value":90},"user":{"ticks":60,"time":{"ms":5}}},"handles":{"limit":{"hard":1024,"soft":1024},"open":39},"info":{"ephemeral_id":"c6282f20-241d-4e69-9762-0633ccb51c4a","uptime":{"ms":60047}},"memstats":{"gc_next":11246624,"memory_alloc":7176432,"memory_total":16269960,"rss":319488},"runtime":{"goroutines":27}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"system":{"load":{"1":0.3,"15":0.1,"5":0.14,"norm":{"1":0.15,"15":0.05,"5":0.07}}}}}} ``` Strangely, its not showing the connection `SUCCESS` event. It should at least log the event when a connection is successfully established to elasticsearch or logstash which ever is set in the output.elasticsearch. ---
**Author:** ![DenizParlak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/denizparlak/32/81179_2.png) [@DenizParlak](https://discuss.elastic.co/u/DenizParlak)\ **Post date:** [December 21, 2020, 10:22am UTC](https://discuss.elastic.co/t/journalbeat-stopped-shipping-logs/259195/3 "2020-12-21T10:22:22Z")
Hi, Same issue on our side with 7.10.0 version. Even we tried different configurations in journalbeat.yml file but every variation has failed and there are no any 'error' messages on logs. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 18, 2021, 12:22pm UTC](https://discuss.elastic.co/t/journalbeat-stopped-shipping-logs/259195/4 "2021-01-18T12:22:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.