Filebeat inptu container rotation json parser error

docker json-file 16K stream buffer split

When log exceeded 16K and log rotation, json parser error

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    output.redis:
      hosts: ["10.102.0.16:6379"]
      key: "dev"
      db: 0
      timeout: 5
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:7.6.0
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: REDIS_HOST
          value: "10.102.0.16"
        - name: REDIS_PORT
          value: "6379"
        - name: KEY
          value: "dev"
        - name: DB
          value: "0"
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 300Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---

Could you please post any logged errors?

filebeat -> redis -> logstash -> elasticsearch

I am through logstash filter json jsonparsefailure

Test a log over 16k

Only happens when the log is over 10M log rotation

4b3038ce270f799222edbca70efbd85b8d36752e42058fd9e45af920963a9701-json.log.1 content
{"log":"testtest...tes","stream":"stderr","time":"2020-05-22T03:35:35.632371865Z"}



4b3038ce270f799222edbca70efbd85b8d36752e42058fd9e45af920963a9701-json.log content
{"log":"ttesttest...test\",\"time\":\"2020-05-22T03:35:35Z\"}\n","stream":"stderr","time":"2020-05-22T03:35:35.632371865Z"}

elasticsearch only see

"_source": {
    "message": "ttesttest...test\",\"time\":\"2020-05-22T03:35:35Z\"}",...

"tags": [
      "_jsonparsefailure"
    ],

@mtojek

fliebeat without any error log

redis list only half log

{"log":"ttesttest...test\",\"time\":\"2020-05-22T03:35:35Z\"}\n","stream":"stderr","time":"2020-05-22T03:35:35.632371865Z"}

I setting it wrong?

Are you sure you JSON log lines are correct, I mean, properly escaped?

@mtojek

I test log is fixed

Only happens when the log is over 10M log rotation

log

{"level":"info","msg":"test","test":"testtest...over 16k omit...testtesttesttest","time":"2020-05-25T09:52:55Z"}

redis

"{\"@timestamp\":\"2020-05-26T02:23:27.336Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"7.6.0\"},\"log\":{\"offset\":1140662,\"file\":{\"path\":\"/test.log\"}},\"message\":\"{\\\"log\\\":\\\"ttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest\\\\\\\",\\\\\\\"time\\\\\\\":\\\\\\\"2020-05-25T09:52:55Z\\\\\\\"}\\\\n\\\",\\\"stream\\\":\\\"stderr\\\",\\\"time\\\":\\\"2020-05-25T09:52:55.735481474Z\\\"}\",\"input\":{\"type\":\"log\"},\"host\":{\"name\":\"5d535347fb0a\"},\"agent\":{\"id\":\"579e046a-09ee-4cc8-91d2-4c82db4ef57e\",\"version\":\"7.6.0\",\"type\":\"filebeat\",\"ephemeral_id\":\"745c5775-ccc0-431d-a8d3-28299ce04ae9\",\"hostname\":\"5d535347fb0a\"},\"ecs\":{\"version\":\"1.4.0\"}}"