High CPU usage after updating filebeat from 7.12.0 to 8.6.2

Elastic Stack Beats
1

After updating to filebeat to 8.6.2 I observe an increase in cpu usage. also tested on 8.6.1 same thing, went back to 8.0.0 and could also observe an increase there, however less than in 8.6.2 and 8.6.1. Is there anything that can explain that?

filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
          add_resource_metadata:
            cronjob: false
            deployment: false
            namespace:
              enabled: true
    fields_under_root: true
    fields:
      kubernetes.cluster: {{ .Values.name }}
      kubernetes.stage: {{ (split "-" .Values.name)._1 }}
    processors:
      - add_host_metadata:
          netinfo.enabled: false
          when.not.equals.kubernetes.namespace_labels.namespace-type: application
      - drop_fields:
          fields: ['ecs.version', 'kubernetes.namespace_uid']
          when.not.equals.kubernetes.namespace_labels.namespace-type: application
      - drop_fields:
          fields: ['kubernetes.node.uid', 'kubernetes.pod.ip', '/^kubernetes.node.labels.*/']
      # the "index-name" field is used by ELK to determine the effective index
      # the effective index is "index-name" suffixed by the current day
      - copy_fields:
          fields:
            - from: kubernetes.labels.logging_acc_k8s_zone/index-name
              to: index-name
          fail_on_error: false
          ignore_missing: true
          when.not.has_fields: ['index-name']
      # all applications in our namespaces will use the acccps-k8s-logs index, if not overwritten by a label
      - add_fields:
          target: ''
          fields:
            index-name: acccps-k8s-logs
          when:
            and:
            - not.has_fields: ['index-name']
            - or:
              - equals.kubernetes.namespace_labels.namespace-type: shared
              - equals.kubernetes.namespace_labels.namespace-type: helper
      - add_fields:
          fields:
            agent.hostname: ${HOSTNAME}
          target: ""
      - copy_fields:
          fields:
            - from: container.image.name
              to: kubernetes.container.image
          target: "kubernetes"
      - decode_json_fields:
          fields: ['message']
          overwrite_keys: true
          target: ""
      # the "tenant" field is just for convinience
      - copy_fields:
          fields:
            - from: kubernetes.namespace_labels.tenant
              to: tenant
          fail_on_error: false
          ignore_missing: true
          when.not.has_fields: ['tenant']
      # drop events without index-name, because ELK can't handle them anyway
      - drop_event:
          when.not.has_fields: ['index-name']
    output.logstash:
      hosts:
      - {{ printf "%s:%d" .Values.log_sink.address (.Values.log_sink.port | int) }}
      ssl:
        certificate_authorities:
          - "/etc/puki-certs/pukirootca1.pem"

above is my config file, when updating to 8.6.2, I drop some fields, add some and copied some see changes below

 - drop_fields:
          fields: ['kubernetes.node.uid', 'kubernetes.pod.ip', '/^kubernetes.node.labels.*/']

      - add_fields:
          fields:
            agent.hostname: ${HOSTNAME}

      - copy_fields:
          fields:
            - from: container.image.name
              to: kubernetes.container.image
          target: "kubernetes"

Tried to comment out those changes to see if they are root cause, but it did not help since the cpu usage was still high.

Any idea why this is happening?

2

I've noticed as well that Filebeat has slowly increased in CPU usage overtime. I'm not 100% sure what causes it, but one of the things I've been looking at recently is switching from the default_config from type: container to type: filestream, as I think part of the "issue" is that in larger scale deployments the total number of files has an impact on Filebeat's performance, and that filestream is in theory supposed to be a much more efficient input type.

I haven't actually had a chance to test this theory yet, so not 100% sure it will make a difference. This comment shows somewhat how to implement the filestream type, but is missing the id value, I think you can do something like:

filebeat.autodiscover:
     providers:
       - type: kubernetes
         node: ${NODE_NAME}
         hints.enabled: true
         hints.default_config:
           type: filestream
           id: kubernetes-container-logs-${kubernetes.pod.name}-${kubernetes.container.id}
           paths:
             - /var/log/containers/*${data.kubernetes.container.id}.log
           parsers:
             - container:
               stream: all
               format: auto

Note: Currently it is not possible to switch from container to filestream without reprocessing logs. This feature appears to be added in 8.7 (https://github.com/elastic/beats/pull/34292)

Another observation, where possible, you have some processors (ex: drop_fields, copy_fields) which don't have ignore_missing: true set to true. I've seen in the past that this can have some negative performance impacts as Filebeat will need to deal with error handling on missing fields.

1 Like
3

Hi BenB196,

switching to filestream seems to solve the issue however with the config you provided above, I can't see any logs in kibana. is there any additional config I have to do to able to view my logs on kibana as usual?

4

Hmm, could you provide the entire config you're using, and do you see any warning or error logs from Filebeat?

5

Below is my config

filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
          add_resource_metadata:
            namespace:
              enabled: true
    fields_under_root: true
    fields:
      kubernetes.cluster: {{ .Values.name }}
      kubernetes.stage: {{ (split "-" .Values.name)._1 }}
    processors:
      - add_host_metadata:
          netinfo.enabled: false
          when.not.equals.kubernetes.namespace_labels.namespace-type: application
      - drop_fields:
          fields: ['ecs.version', 'kubernetes.namespace_uid']
          when.not.equals.kubernetes.namespace_labels.namespace-type: application
      # the "index-name" field is used by ELK to determine the effective index
      # the effective index is "index-name" suffixed by the current day
      - copy_fields:
          fields:
            - from: kubernetes.labels.logging_acc_k8s_zone/index-name
              to: index-name
          fail_on_error: false
          ignore_missing: true
          when.not.has_fields: ['index-name']
      # all applications in our namespaces will use the acccps-k8s-logs index, if not overwritten by a label
      - add_fields:
          target: ''
          fields:
            index-name: acccps-k8s-logs
          when:
            and:
            - not.has_fields: ['index-name']
            - or:
              - equals.kubernetes.namespace_labels.namespace-type: shared
              - equals.kubernetes.namespace_labels.namespace-type: helper
      - decode_json_fields:
          fields: ['message']
          add_error_key: true
          overwrite_keys: true
          target: ""
      # the "tenant" field is just for convinience
      - copy_fields:
          fields:
            - from: kubernetes.namespace_labels.tenant
              to: tenant
          fail_on_error: false
          ignore_missing: true
          when.not.has_fields: ['tenant']
      # drop events without index-name, because ELK can't handle them anyway
      - drop_event:
          when.not.has_fields: ['index-name']
    output.logstash:
      hosts:
      - {{ printf "%s:%d" .Values.log_sink.address (.Values.log_sink.port | int) }}
      ssl:
        certificate_authorities:
          - "/etc/puki-certs/pukirootca1.pem"

As of the logs, I have the message below repeatedly

rrent":0}},"system":{"load":{"1":2,"15":1.09,"5":1.36,"norm":{"1":0.25,"15":0.1363,"5":0.17}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-09T12:01:18.382Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"stats":{"periods":53,"throttled":{"ns":106306411,"periods":3}}},"cpuacct":{"total":{"ns":1020256750}},"memory":{"mem":{"usage":{"bytes":62091264}}}},"cpu":{"system":{"ticks":22330,"time":{"ms":60}},"total":{"ticks":467820,"time":{"ms":1020},"value":467820},"user":{"ticks":445490,"time":{"ms":960}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"0bb553cf-dfc8-4e41-9cda-ed1d68eb89b8","uptime":{"ms":15030062},"version":"8.6.2"},"memstats":{"gc_next":24686296,"memory_alloc":20083784,"memory_total":86494130392,"rss":133447680},"runtime":{"goroutines":54}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":1.7,"15":1.09,"5":1.35,"norm":{"1":0.2125,"15":0.1363,"5":0.1688}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-09T12:01:48.379Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"stats":{"periods":37,"throttled":{"ns":66430629,"periods":3}}},"cpuacct":{"total":{"ns":987994974}},"memory":{"mem":{"usage":{"bytes":64339968}}}},"cpu":{"system":{"ticks":22380,"time":{"ms":50}},"total":{"ticks":468820,"time":{"ms":1000},"value":468820},"user":{"ticks":446440,"time":{"ms":950}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"0bb553cf-dfc8-4e41-9cda-ed1d68eb89b8","uptime":{"ms":15060062},"version":"8.6.2"},"memstats":{"gc_next":24648440,"memory_alloc":20892144,"memory_total":86677212456,"rss":135348224},"runtime":{"goroutines":54}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":1.48,"15":1.09,"5":1.33,"norm":{"1":0.185,"15":0.1363,"5":0.1663}}}},"ecs.version":"1.6.0"}}
6

Daemonset

{{- range $type, $data := .Values.addons.filebeat }}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat-{{ $type }}
  namespace: logging
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
        # TODO Add component, version labels
    spec:
      priorityClassName: "cluster-essential"
      tolerations:
        - key: node.kubernetes.io/role
          value: master
          effect: "NoSchedule"
        - key: "CriticalAddonsOnly"
          operator: "Exists"
      nodeSelector:
        node.kubernetes.io/role: {{ $type }}
      serviceAccountName: filebeat-platform
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:{{ $.Values.sw_versions.filebeat.version }}
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
          privileged: true
        resources:
        {{ toYaml $data.resources | nindent 12 }}
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: config
          mountPath: /etc/fill_index_fallback_processor.js
          readOnly: true
          subPath: fill_index_fallback_processor.js
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
        - name: ca-certificates
          mountPath: /etc/puki-certs
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
      - name: ca-certificates
        configMap:
          name: ca-certificates
{{- end }}
7

Sorry, I meant could you provide the configuration you tried with type: filestream. This appears to be the config with type: container.

8

with filestream I just used what you proposed and nothing more

filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: filestream
            id: kubernetes-container-logs-${kubernetes.pod.name}-${kubernetes.container.id}
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
            parsers:
              - container:
                stream: all
                format: auto
          add_resource_metadata:
            cronjob: false
            deployment: false
            namespace:
              enabled: true
    fields_under_root: true
    fields:
      kubernetes.cluster: {{ .Values.name }}
      kubernetes.stage: {{ (split "-" .Values.name)._1 }}
    processors:
      - add_host_metadata:
          netinfo.enabled: false
          when.not.equals.kubernetes.namespace_labels.namespace-type: application
      - drop_fields:
          fields: ['ecs.version', 'kubernetes.namespace_uid']
          ignore_missing: true
          when.not.equals.kubernetes.namespace_labels.namespace-type: application
      - drop_fields:
          fields: ['kubernetes.node.uid', 'kubernetes.pod.ip', '/^kubernetes.node.labels.*/']
          ignore_missing: true
      # the "index-name" field is used by ELK to determine the effective index
      # the effective index is "index-name" suffixed by the current day
      - copy_fields:
          fields:
            - from: kubernetes.labels.logging_acc_k8s_zone/index-name
              to: index-name
          fail_on_error: false
          ignore_missing: true
          when.not.has_fields: ['index-name']
      # all applications in our namespaces will use the acccps-k8s-logs index, if not overwritten by a label
      - add_fields:
          target: ''
          fields:
            index-name: acccps-k8s-logs
          when:
            and:
            - not.has_fields: ['index-name']
            - or:
              - equals.kubernetes.namespace_labels.namespace-type: shared
              - equals.kubernetes.namespace_labels.namespace-type: helper
      - add_fields:
          fields:
            agent.hostname: ${HOSTNAME}
          target: ""
      - copy_fields:
          fields:
            - from: container.image.name
              to: kubernetes.container.image
          fail_on_error: false
          ignore_missing: true
          target: "kubernetes"
      - decode_json_fields:
          fields: ['message']
          overwrite_keys: true
          target: ""
      # the "tenant" field is just for convinience
      - copy_fields:
          fields:
            - from: kubernetes.namespace_labels.tenant
              to: tenant
          fail_on_error: false
          ignore_missing: true
          when.not.has_fields: ['tenant']
      # drop events without index-name, because ELK can't handle them anyway
      - drop_event:
          when.not.has_fields: ['index-name']
    output.logstash:
      hosts:
      - {{ printf "%s:%d" .Values.log_sink.address (.Values.log_sink.port | int) }}
      ssl:
        certificate_authorities:
          - "/etc/puki-certs/pukirootca1.pem"
9

That config looks correct, do you see anything in the Filebeat logs that would highlight any issues?

10

not at all

11

Hi @BenB196 ,

when I compare the logs from 8.6.2 using filestream an that of 7.12.0 using container, I can see that both goes smoothly till the point where you have starting auto discover manager. from there in 7.12 the config path is set and the harvester start harvesting logs, however for the 8.6.2 config with filestream no haverster start haversting see below

7.12 logs 

2023-03-13T07:33:08.440Z	INFO	instance/beat.go:660	Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2023-03-13T07:33:08.548Z	INFO	instance/beat.go:668	Beat ID: fd5efd2d-7735-4205-99bb-75bd95ba731a
2023-03-13T07:33:08.552Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2023-03-13T07:33:08.552Z	INFO	[beat]	instance/beat.go:996	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "fd5efd2d-7735-4205-99bb-75bd95ba731a"}}}
2023-03-13T07:33:08.552Z	INFO	[beat]	instance/beat.go:1005	Build info	{"system_info": {"build": {"commit": "08e20483a651ea5ad60115f68ff0e53e6360573a", "libbeat": "7.12.0", "time": "2021-03-18T06:16:51.000Z", "version": "7.12.0"}}}
2023-03-13T07:33:08.552Z	INFO	[beat]	instance/beat.go:1008	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":16,"version":"go1.15.8"}}}
2023-03-13T07:33:08.554Z	INFO	[beat]	instance/beat.go:1012	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2023-03-13T07:26:06Z","containerized":true,"name":"acc-kworker-be-lab-iz2-bap003","ip":["127.0.0.1/8","10.234.48.136/26","192.168.122.1/23","100.69.235.0/32"],"kernel_version":"5.15.89-flatcar","mac":["34:48:ed:e9:78:38","34:48:ed:e9:78:39","4c:d9:8f:a2:a3:4b","4c:d9:8f:a2:a3:4c","02:42:9a:92:29:df","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee"],"os":{"type":"linux","family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":9,"patch":2009,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"44ccc339d95bd51386fcfc5d8f041927"}}}
2023-03-13T07:33:08.554Z	INFO	[beat]	instance/beat.go:1041	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 7, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2023-03-13T07:33:07.390Z"}}}
2023-03-13T07:33:08.554Z	INFO	instance/beat.go:304	Setup Beat: filebeat; Version: 7.12.0
2023-03-13T07:33:08.554Z	WARN	[cfgwarn]	tlscommon/config.go:101	DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0
2023-03-13T07:33:08.555Z	INFO	[publisher]	pipeline/module.go:113	Beat name: acc-kworker-be-lab-iz2-bap003
2023-03-13T07:33:08.556Z	WARN	beater/filebeat.go:178	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2023-03-13T07:33:08.556Z	INFO	[monitoring]	log/log.go:117	Starting metrics logging every 30s
2023-03-13T07:33:08.556Z	INFO	instance/beat.go:468	filebeat start running.
2023-03-13T07:33:08.631Z	INFO	memlog/store.go:119	Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2023-03-13T07:33:08.631Z	INFO	memlog/store.go:124	Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0
2023-03-13T07:33:08.631Z	WARN	beater/filebeat.go:381	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2023-03-13T07:33:08.632Z	INFO	[registrar]	registrar/registrar.go:109	States Loaded from registrar: 0
2023-03-13T07:33:08.632Z	INFO	[crawler]	beater/crawler.go:71	Loading Inputs: 0
2023-03-13T07:33:08.632Z	INFO	[crawler]	beater/crawler.go:108	Loading and starting Inputs completed. Enabled inputs: 0
2023-03-13T07:33:08.633Z	INFO	[autodiscover.pod]	kubernetes/util.go:99	kubernetes: Using node acc-kworker-be-lab-iz2-bap003 provided in the config
2023-03-13T07:33:08.633Z	INFO	[autodiscover]	autodiscover/autodiscover.go:113	Starting autodiscover manager
2023-03-13T07:33:08.942Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*9ed9dfce19b2cf82974cbb1b77dedb43b6394e7e21b5f2b35dfac4720a11e9dd.log]
2023-03-13T07:33:08.943Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*9ed9dfce19b2cf82974cbb1b77dedb43b6394e7e21b5f2b35dfac4720a11e9dd.log]
2023-03-13T07:33:08.943Z	INFO	log/input.go:157	Configured paths: [/var/log/containers/*8b8004ec0eafb9bb487e0802d60bec5e84f9dd0b18b06b17be8051bc62f7afd4.log]
2023-03-13T07:33:08.944Z	INFO	log/harvester.go:302	Harvester started for file: /var/log/containers/kubernetes-metrics-scraper-65cd4674d-j6jkp_kube-system_kubernetes-metrics-scraper-9ed9dfce19b2cf82974cbb1b77dedb43b6394e7e21b5f2b35dfac4720a11e9dd.log

8.6.2 with filestream logs

{"log.level":"info","@timestamp":"2023-03-13T08:13:44.395Z","log.origin":{"file.name":"instance/beat.go","file.line":724},"message":"Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-13T08:13:44.395Z","log.origin":{"file.name":"instance/beat.go","file.line":732},"message":"Beat ID: 51f0802f-e467-4548-b71b-e7e01344bdb2","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-13T08:13:44.398Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-13T08:13:44.398Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1096},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/usr/share/filebeat","data":"/usr/share/filebeat/data","home":"/usr/share/filebeat","logs":"/usr/share/filebeat/logs"},"type":"filebeat","uuid":"51f0802f-e467-4548-b71b-e7e01344bdb2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-13T08:13:44.398Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1105},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"9b77c2c135c228c2eedc310f6e975bb1a76169b1","libbeat":"8.6.2","time":"2023-02-12T04:37:19.000Z","version":"8.6.2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-13T08:13:44.398Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1108},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":16,"version":"go1.18.10"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-13T08:13:44.399Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1112},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-03-12T15:31:47Z","containerized":true,"name":"acc-kworker-be-lab-iz1-bs001","ip":["127.0.0.1/8","10.234.32.78/26","192.168.122.1/23","100.66.139.0/32"],"kernel_version":"5.15.89-flatcar","mac":["4c:d9:8f:a2:a7:8b","4c:d9:8f:a2:a7:8c","f4:02:70:fa:7d:fc","f4:02:70:fa:7d:fd","02:42:dc:95:a5:2a","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee","ee:ee:ee:ee:ee:ee"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.5 LTS (Focal Fossa)","major":20,"minor":4,"patch":5,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-13T08:13:44.399Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1141},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null},"cwd":"/usr/share/filebeat","exe":"/usr/share/filebeat/filebeat","name":"filebeat","pid":7,"ppid":1,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2023-03-13T08:13:44.260Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-03-13T08:13:44.400Z","log.origin":{"file.name":"instance/beat.go","file.line":296},"message":"Setup Beat: filebeat; Version: 8.6.2","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-03-13T08:13:44.455Z","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-13T08:13:44.455Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: acc-kworker-be-lab-iz1-bs001","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-13T08:13:44.455Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-03-13T08:13:44.455Z","log.origin":{"file.name":"beater/filebeat.go","file.line":164},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-13T08:13:44.456Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":145},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-13T08:13:44.456Z","log.origin":{"file.name":"instance/beat.go","file.line":486},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-13T08:13:44.457Z","log.origin":{"file.name":"memlog/store.go","file.line":127},"message":"Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=4655471","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-13T08:13:44.496Z","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=4658584","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-03-13T08:13:44.496Z","log.origin":{"file.name":"beater/filebeat.go","file.line":290},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-13T08:13:44.496Z","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 65","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-13T08:13:44.496Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-13T08:13:44.496Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-13T08:13:44.497Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-13T08:13:44.497Z","log.logger":"autodiscover.pod","log.origin":{"file.name":"kubernetes/util.go","file.line":122},"message":"kubernetes: Using node acc-kworker-be-lab-iz1-bs001 provided in the config","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-13T08:13:44.508Z","log.logger":"autodiscover","log.origin":{"file.name":"autodiscover/autodiscover.go","file.line":118},"message":"Starting autodiscover manager","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-13T08:14:14.458Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":100000}},"id":"/","stats":{"periods":49,"throttled":{"ns":247563927,"periods":2}}},"cpuacct":{"id":"/","total":{"ns":1153578084}},"memory":{"id":"/","mem":{"limit":{"bytes":4294967296},"usage":{"bytes":69206016}}}},"cpu":{"system":{"ticks":70,"time":{"ms":70}},"total":{"ticks":1120,"time":{"ms":1120},"value":1120},"user":{"ticks":1050,"time":{"ms":1050}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"9607da3d-a753-43cf-8f79-cc8cd46e7456","name":"filebeat","uptime":{"ms":30102},"version":"8.6.2"},"memstats":{"gc_next":33430536,"memory_alloc":30300792,"memory_sys":54019080,"memory_total":298627032,"rss":135970816},"runtime":{"goroutines":54}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":16},"load":{"1":0.6,"15":0.45,"5":0.39,"norm":{"1":0.0375,"15":0.0281,"5":0.0244}}}},"ecs.version":"1.6.0"}}
12

Hmm. something I'm just noticing from the example I copied, is that it has invalid yaml.

parsers:
  - container:
    stream: all
    format: auto

Should instead be:

parsers:
  - container:
      stream: all
      format: auto

Where stream and format are under container

13

didn't also pay attention to that. However it doesn't fix the issue. the logs are still the same

14

Any other suggestion here?

15

Hmm, the only other thing I can think of is try enabling debug logging, this will hopefully show something that might be useful for why this isn't working as intended.