JournalBeat Stopped Shipping Logs

JournalBeat stopped shipping logs on all environments for us from 19th Dec, 2020. We configured our beats services to ship logs from aws instance to elasticsearch through logstash. Our filebeat and metricbeat logs are being shipped fine. It has issue only shipping the journalbeat logs.

We have verified the SELinuxOptions as well. We tried enabling the debug logs on elasticsearch, logstash and journalbeat as well. Not finding any errors but still no logs in the elasticsearch.

ELK Stack version: 7.10.1
JournalBeats version: 7.10.1
Kubernetes version: 1.19.4

Below is our kubernetes manifest for the journalbeat.

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
  name: journalbeat
  namespace: kube-system
spec:
  allowedCapabilities:
    - KILL
    - CHOWN
    - FSETID
    - FOWNER
    - SETGID
    - SETUID
    - SETFCAP
    - SETPCAP
    - AUDIT_WRITE
    - NET_BIND_SERVICE
  fsGroup:
    rule: RunAsAny
  hostIPC: false
  hostNetwork: false
  hostPID: false
  privileged: false
  requiredDropCapabilities:
    - MKNOD
    - DAC_OVERRIDE
    - NET_RAW
    - SYS_CHROOT
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
    - secret
    - configMap
    - hostPath
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: journalbeat
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: journalbeat-config
  namespace: kube-system
  labels:
    k8s-app: journalbeat-logging
    version: v1
data:
  journalbeat.yml: |
    name: "${NODENAME}"
    journalbeat.inputs:
    - paths: []
      seek: cursor
      cursor_seek_fallback: tail

    processors:
    - add_kubernetes_metadata:
        host: "${NODENAME}"
        in_cluster: true
        default_indexers.enabled: false
        default_matchers.enabled: false
        indexers:
          - container:
        matchers:
          - fields:
              lookup_fields: ["container.id"]
    - decode_json_fields:
        fields: ["message"]
        process_array: false
        max_depth: 1
        target: ""
        overwrite_keys: true
    - drop_event.when:
        or:
        - regexp.kubernetes.pod.name: "filebeat-.*"
        - regexp.kubernetes.pod.name: "journalbeat-.*"
        - regexp.kubernetes.pod.name: "metrics-server-.*"
        - equals.syslog.identifier: "audit"
        - regexp.message: '^.*?\baudit\b.*?$'

        - and:
          - equals.systemd.unit: "docker.service"
          - or:
            - regexp.message: '^.*?\bJournalbeat is experimental.*?$'

        - and:
          - equals.systemd.unit: "init.scope"
          - or:
            - regexp.message: '^.*?\bStarted libcontainer container.*?$'
            - equals.message: "Started rpm-ostree System Management Daemon."
            - regexp.message: '^.*?\bSucceeded.*?$'

    # Updated to ignore conflict of indices between environments
    setup.ilm.enabled: false
    setup.template.enabled: false

    output.logstash:
      hosts: '${LOGSTASH_HOSTS}'
      compression_level: 7
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: journalbeat
rules:
  - apiGroups:
      - ""
    resources:
      - namespaces
      - pods
    verbs:
      - get
      - watch
      - list
  - apiGroups:
      - extensions
    resourceNames:
      - journalbeat
    resources:
      - podsecuritypolicies
    verbs:
      - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: journalbeat
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: journalbeat
subjects:
  - kind: ServiceAccount
    name: journalbeat
    namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: journalbeat
  namespace: kube-system
  labels:
    k8s-app: journalbeat-logging
    version: v1
spec:
  selector:
    matchLabels:
      k8s-app: journalbeat-logging
      version: v1
  template:
    metadata:
      labels:
        k8s-app: journalbeat-logging
        version: v1
        app: journalbeat
        name: journalbeat
    spec:
      containers:
        - args:
            - -e
            - -c
            - /etc/journalbeat.yml
          command:
            - journalbeat
          env:
            - name: NODENAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: PODNAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: LOGSTASH_HOSTS
              value: logstash.company.com
            - name: LOGSTASH_PORT
              value: "5044"
            - name: ENVIRONMENT
              value: prod

          image: docker.elastic.co/beats/journalbeat:7.10.1
          imagePullPolicy: Always
          name: journalbeat
          resources:
            limits:
              cpu: 600m
              memory: 800Mi
            requests:
              cpu: 200m
              memory: 400Mi
          volumeMounts:
            - mountPath: /usr/share/journalbeat/data
              name: data
            - mountPath: /var/log/journal
              name: var-journal
            - mountPath: /run/log/journal
              name: run-journal
            - mountPath: /etc/journalbeat.yml
              name: config
              subPath: journalbeat.yml
            - mountPath: /etc/machine-id
              name: machine-id
      hostNetwork: true
      nodeSelector: {}

      securityContext:
        seLinuxOptions:
          user: system_u
          role: system_r
          type: spc_t
          level: s0

        fsGroup: 0
        runAsUser: 0
      serviceAccountName: journalbeat
      terminationGracePeriodSeconds: 60
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
      volumes:
        - hostPath:
            path: /var/log/journal/journalbeat-data
          name: data
        - hostPath:
            path: /var/log/journal
          name: var-journal
        - hostPath:
            path: /run/log/journal
          name: run-journal
        - hostPath:
            path: /etc/machine-id
          name: machine-id
        - configMap:
            items:
              - key: journalbeat.yml
                path: journalbeat.yml
            name: journalbeat-config
          name: config
1 Like

We are facing the same issue.
The following are the logs in DEBUG mode.

DEBUG OUTPUT

$ kubectl logs -f journalbeat-9852r -n kube-system
2020-12-20T19:54:08.609Z        INFO    instance/beat.go:645    Home path: [/usr/share/journalbeat] Config path: [/usr/share/journalbeat] Data path: [/usr/share/journalbeat/data] Logs path: [/usr/share/journalbeat/logs]
2020-12-20T19:54:08.609Z        DEBUG   [beat]  instance/beat.go:697    Beat metadata path: /usr/share/journalbeat/data/meta.json
2020-12-20T19:54:08.609Z        INFO    instance/beat.go:653    Beat ID: 905e574e-0920-4b40-ab17-055d1bc2a1ba
2020-12-20T19:54:08.610Z        DEBUG   [conditions]    conditions/conditions.go:98     New condition regexp: map[]
2020-12-20T19:54:08.611Z        DEBUG   [conditions]    conditions/conditions.go:98     New condition equals: map[message:{0 Started rpm-ostree System Management Daemon. false}]
2020-12-20T19:54:08.611Z        DEBUG   [conditions]    conditions/conditions.go:98     New condition regexp: map[]
2020-12-20T19:54:08.612Z        DEBUG   [conditions]    conditions/conditions.go:98     New condition equals: map[systemd.unit:{0 init.scope false}] and regexp: map[] or equals: map[message:{0 Started rpm-ostree System Management Daemon. false}] or regexp: map[]
2020-12-20T19:54:08.612Z        DEBUG   [conditions]    conditions/conditions.go:98     New condition regexp: map[] or regexp: map[] or regexp: map[] or equals: map[syslog.identifier:{0 audit false}] or regexp: map[] or equals: map[systemd.unit:{0 docker.service false}] and regexp: map[] or equals: map[systemd.unit:{0 init.scope false}] and regexp: map[] or equals: map[message:{0 Started rpm-ostree System Management Daemon. false}] or regexp: map[]
2020-12-20T19:54:08.612Z        DEBUG   [processors]    processors/processor.go:120     Generated new processors: add_kubernetes_metadata, decode_json_fields=message, drop_event, condition=regexp: map[] or regexp: map[] or regexp: map[] or equals: map[syslog.identifier:{0 audit false}] or regexp: map[] or equals: map[systemd.unit:{0 docker.service false}] and regexp: map[] or equals: map[systemd.unit:{0 init.scope false}] and regexp: map[] or equals: map[message:{0 Started rpm-ostree System Management Daemon. false}] or regexp: map[]
2020-12-20T19:54:08.612Z        DEBUG   [seccomp]       seccomp/seccomp.go:117  Loading syscall filter  {"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","chown","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchmodat","fchown","fchownat","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}}
2020-12-20T19:54:08.612Z        INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2020-12-20T19:54:08.612Z        INFO    [beat]  instance/beat.go:981    Beat info       {"system_info": {"beat": {"path": {"config": "/usr/share/journalbeat", "data": "/usr/share/journalbeat/data", "home": "/usr/share/journalbeat", "logs": "/usr/share/journalbeat/logs"}, "type": "journalbeat", "uuid": "905e574e-0920-4b40-ab17-055d1bc2a1ba"}}}
2020-12-20T19:54:08.612Z        INFO    [beat]  instance/beat.go:993    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.14.7"}}}
2020-12-20T19:54:08.614Z        INFO    [beat]  instance/beat.go:997    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-12-17T20:04:23Z","containerized":true,"name":"ip-10-134-109-2","ip":["127.0.0.1/8","::1/128","10.134.109.2/18","2a05:d018:59e:5801:5e33:4db8:622e:4e6a/128","fe80::14:b9ff:fe99:d8ef/64","172.17.0.1/16","10.136.24.142/32","10.136.175.78/32","10.136.241.110/32","10.136.83.61/32","10.136.52.87/32","10.136.105.175/32","10.136.135.51/32","10.136.27.74/32","10.136.184.145/32","10.136.220.231/32","10.136.80.253/32","10.136.26.147/32","10.136.235.176/32","10.136.93.201/32","10.136.35.104/32","10.136.26.200/32","10.136.46.245/32","10.136.156.12/32","10.136.0.10/32","10.136.198.82/32","10.136.51.141/32","10.136.48.143/32","10.136.234.155/32","10.136.195.0/32","10.136.37.104/32","10.136.143.231/32","10.136.210.147/32","10.136.108.226/32","10.136.91.20/32","10.136.0.1/32","10.136.8.225/32","10.136.148.246/32","10.136.102.34/32","10.136.127.140/32","10.136.220.125/32","10.136.237.12/32","10.2.26.0/32","fe80::90e5:52ff:fec7:eed9/64","10.2.26.1/24","fe80::2021:c4ff:fef6:b1d1/64","fe80::1405:d6ff:fedd:b417/64","fe80::1c1e:2ff:fe63:3f4b/64","fe80::dc01:5ff:fef4:f2d3/64","fe80::2035:2dff:fe1f:f79e/64","fe80::7036:3eff:feaa:7203/64","fe80::c0b1:96ff:fed9:338f/64","fe80::84f7:d5ff:febe:58c2/64","fe80::8493:6cff:fe11:28b9/64"],"kernel_version":"5.9.11-200.fc33.x86_64","mac":["02:14:b9:99:d8:ef","02:42:d6:af:81:eb","9a:8f:68:58:ef:1b","92:e5:52:c7:ee:d9","22:21:c4:f6:b1:d1","fa:91:26:19:8d:f6","56:f8:e8:45:61:64","ce:92:e7:07:c2:50","ca:a8:8c:bc:85:5e","1a:f9:45:c3:3c:1e","56:c8:1d:a3:81:a2","72:9e:6c:a2:ee:7c","ae:30:e2:f7:39:62"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"ec2756fb725f5dcf7bbf13bc22225dd8"}}}
2020-12-20T19:54:08.614Z        INFO    [beat]  instance/beat.go:1026   Process info    {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/journalbeat", "exe": "/usr/share/journalbeat/journalbeat", "name": "journalbeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-12-20T19:54:08.510Z"}}}
2020-12-20T19:54:08.615Z        INFO    instance/beat.go:299    Setup Beat: journalbeat; Version: 7.10.0
2020-12-20T19:54:08.615Z        DEBUG   [beat]  instance/beat.go:325    Initializing output plugins
2020-12-20T19:54:08.615Z        DEBUG   [publisher]     pipeline/consumer.go:148        start pipeline event consumer
2020-12-20T19:54:08.615Z        INFO    [publisher]     pipeline/module.go:113  Beat name: ip-10-134-109-2
2020-12-20T19:54:08.615Z        WARN    [cfgwarn]       beater/journalbeat.go:55        EXPERIMENTAL: Journalbeat is experimental.
2020-12-20T19:54:08.616Z        INFO    add_kubernetes_metadata/kubernetes.go:71        add_kubernetes_metadata: kubernetes env detected, with version: v1.19.3
2020-12-20T19:54:08.616Z        INFO    [kubernetes]    kubernetes/util.go:99   kubernetes: Using node ip-10-134-109-2 provided in the config   {"libbeat.processor": "add_kubernetes_metadata"}
2020-12-20T19:54:08.616Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:162       Initializing a new Kubernetes watcher using host: ip-10-134-109-2       {"libbeat.processor": "add_kubernetes_metadata"}
2020-12-20T19:54:08.633Z        DEBUG   [input] reader/journal.go:75    New journal is opened for reading       {"path": "LOCAL_SYSTEM_JOURNAL"}
2020-12-20T19:54:08.633Z        DEBUG   [input] input/input.go:112      New input is created for paths []
2020-12-20T19:54:08.633Z        INFO    instance/beat.go:455    journalbeat start running.
2020-12-20T19:54:08.633Z        INFO    [journalbeat]   beater/journalbeat.go:93        journalbeat is running! Hit CTRL-C to stop it.
2020-12-20T19:54:08.638Z        INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2020-12-20T19:54:08.729Z        DEBUG   [kubernetes]    kubernetes/watcher.go:165       cache sync done
2020-12-20T19:54:08.729Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:182       Adding kubernetes pod: frontend/application-www-spot-pod-6b9dcd499b-shg4t       {"libbeat.processor": "add_kubernetes_metadata"}
2020-12-20T19:54:08.729Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:258       Created index e8736a2dde3b9310b20f6feada00848ae72e6b4aa27fdf4fd34bc2e7dcf8a30d for pod frontend/application-www-spot-pod-6b9dcd499b-shg4t       {"libbeat.processor": "add_kubernetes_metadata"}
2020-12-20T19:54:08.729Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:182       Adding kubernetes pod: frontend/application-backoffice-spot-pod-799d4c7556-6cs7b        {"libbeat.processor": "add_kubernetes_metadata"}
2020-12-20T19:54:08.729Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:258       Created index 97372684e4b697c6040a2abcf4dfefbda2e15c3022d0c0a0719f3e6fa7de7c9f for pod frontend/application-backoffice-spot-pod-799d4c7556-6cs7b        {"libbeat.processor": "add_kubernetes_metadata"}
2020-12-20T19:54:08.818Z        DEBUG   [kubernetes]    add_kubernetes_metadata/kubernetes.go:258       Created index cedef08e8b58bdbb172ec223745e038b574a5f744d6ff6d1765feb9d1d50624b for pod kube-system/journalbeat-9852r    {"libbeat.processor": "add_kubernetes_metadata"}
2020-12-20T19:54:38.640Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"id":"docker-cedef08e8b58bdbb172ec223745e038b574a5f744d6ff6d1765feb9d1d50624b.scope"},"cpuacct":{"id":"docker-cedef08e8b58bdbb172ec223745e038b574a5f744d6ff6d1765feb9d1d50624b.scope"},"memory":{"id":"docker-cedef08e8b58bdbb172ec223745e038b574a5f744d6ff6d1765feb9d1d50624b.scope"}},"cpu":{"system":{"ticks":30,"time":{"ms":32}},"total":{"ticks":80,"time":{"ms":88},"value":80},"user":{"ticks":50,"time":{"ms":56}}},"handles":{"limit":{"hard":1024,"soft":1024},"open":39},"info":{"ephemeral_id":"c6282f20-241d-4e69-9762-0633ccb51c4a","uptime":{"ms":30047}},"memstats":{"gc_next":11246624,"memory_alloc":6747480,"memory_total":15841008,"rss":73404416},"runtime":{"goroutines":27}},"journalbeat":{"journals":{"journal_0":{"path":"LOCAL_SYSTEM_JOURNAL","size_in_bytes":3816935424}}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":0}}},"system":{"cpu":{"cores":2},"load":{"1":0.01,"15":0.08,"5":0.08,"norm":{"1":0.005,"15":0.04,"5":0.04}}}}}}
2020-12-20T19:55:08.640Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":30,"time":{"ms":5}},"total":{"ticks":90,"time":{"ms":10},"value":90},"user":{"ticks":60,"time":{"ms":5}}},"handles":{"limit":{"hard":1024,"soft":1024},"open":39},"info":{"ephemeral_id":"c6282f20-241d-4e69-9762-0633ccb51c4a","uptime":{"ms":60047}},"memstats":{"gc_next":11246624,"memory_alloc":7176432,"memory_total":16269960,"rss":319488},"runtime":{"goroutines":27}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"system":{"load":{"1":0.3,"15":0.1,"5":0.14,"norm":{"1":0.15,"15":0.05,"5":0.07}}}}}}

Strangely, its not showing the connection SUCCESS event. It should at least log the event when a connection is successfully established to elasticsearch or logstash which ever is set in the output.elasticsearch.

Hi,

Same issue on our side with 7.10.0 version. Even we tried different configurations in journalbeat.yml file but every variation has failed and there are no any 'error' messages on logs.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.