# Filebeat inptu container rotation json parser error
**URL:** https://discuss.elastic.co/t/filebeat-inptu-container-rotation-json-parser-error/233713
**Category:** Beats
**Tags:** docker, filebeat
**Created:** [May 21, 2020, 9:51am UTC](https://discuss.elastic.co/t/filebeat-inptu-container-rotation-json-parser-error/233713 "2020-05-21T09:51:13Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@suoititcif](https://discuss.elastic.co/u/suoititcif)
#### Post date: [May 21, 2020, 9:51am UTC](https://discuss.elastic.co/t/filebeat-inptu-container-rotation-json-parser-error/233713/1 "2020-05-21T09:51:13Z")
docker json-file 16K stream buffer split
When log exceeded 16K and log rotation, json parser error
```auto
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
processors:
- add_cloud_metadata:
- add_host_metadata:
output.redis:
hosts: ["10.102.0.16:6379"]
key: "dev"
db: 0
timeout: 5
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: filebeat
namespace: kube-system
labels:
k8s-app: filebeat
spec:
selector:
matchLabels:
k8s-app: filebeat
template:
metadata:
labels:
k8s-app: filebeat
spec:
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat:7.6.0
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
env:
- name: REDIS_HOST
value: "10.102.0.16"
- name: REDIS_PORT
value: "6379"
- name: KEY
value: "dev"
- name: DB
value: "0"
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
runAsUser: 0
# If using Red Hat OpenShift uncomment this:
#privileged: true
resources:
limits:
memory: 300Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
- name: data
mountPath: /usr/share/filebeat/data
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: varlog
mountPath: /var/log
readOnly: true
volumes:
- name: config
configMap:
defaultMode: 0600
name: filebeat-config
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: varlog
hostPath:
path: /var/log
# data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
- name: data
hostPath:
path: /var/lib/filebeat-data
type: DirectoryOrCreate
---
```
---
### Author:  [@mtojek](https://discuss.elastic.co/u/mtojek)
#### Post date: [May 21, 2020, 1:34pm UTC](https://discuss.elastic.co/t/filebeat-inptu-container-rotation-json-parser-error/233713/2 "2020-05-21T13:34:56Z")
Could you please post any logged errors?
---
### Author:  [@suoititcif](https://discuss.elastic.co/u/suoititcif)
#### Post date: [May 22, 2020, 3:57am UTC](https://discuss.elastic.co/t/filebeat-inptu-container-rotation-json-parser-error/233713/3 "2020-05-22T03:57:26Z")
filebeat -\> redis -\> logstash -\> elasticsearch
I am through logstash filter json jsonparsefailure
Test a log over 16k
Only happens when the log is over 10M log rotation
```auto
4b3038ce270f799222edbca70efbd85b8d36752e42058fd9e45af920963a9701-json.log.1 content
{"log":"testtest...tes","stream":"stderr","time":"2020-05-22T03:35:35.632371865Z"}
4b3038ce270f799222edbca70efbd85b8d36752e42058fd9e45af920963a9701-json.log content
{"log":"ttesttest...test\",\"time\":\"2020-05-22T03:35:35Z\"}\n","stream":"stderr","time":"2020-05-22T03:35:35.632371865Z"}
```
elasticsearch only see
```auto
"_source": {
"message": "ttesttest...test\",\"time\":\"2020-05-22T03:35:35Z\"}",...
"tags": [
"_jsonparsefailure"
],
```
---
### Author:  [@suoititcif](https://discuss.elastic.co/u/suoititcif)
#### Post date: [May 25, 2020, 9:55am UTC](https://discuss.elastic.co/t/filebeat-inptu-container-rotation-json-parser-error/233713/5 "2020-05-25T09:55:17Z")
@mtojek
fliebeat without any error log
redis list only half log
```auto
{"log":"ttesttest...test\",\"time\":\"2020-05-22T03:35:35Z\"}\n","stream":"stderr","time":"2020-05-22T03:35:35.632371865Z"}
```
I setting it wrong?
---
### Author:  [@mtojek](https://discuss.elastic.co/u/mtojek)
#### Post date: [May 25, 2020, 12:51pm UTC](https://discuss.elastic.co/t/filebeat-inptu-container-rotation-json-parser-error/233713/6 "2020-05-25T12:51:48Z")
Are you sure you JSON log lines are correct, I mean, properly escaped?
---
### Author:  [@suoititcif](https://discuss.elastic.co/u/suoititcif)
#### Post date: [May 26, 2020, 2:42am UTC](https://discuss.elastic.co/t/filebeat-inptu-container-rotation-json-parser-error/233713/7 "2020-05-26T02:42:26Z")
@mtojek
I test log is fixed
Only happens when the log is over 10M log rotation
log
```auto
{"level":"info","msg":"test","test":"testtest...over 16k omit...testtesttesttest","time":"2020-05-25T09:52:55Z"}
```
redis
```auto
"{\"@timestamp\":\"2020-05-26T02:23:27.336Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"7.6.0\"},\"log\":{\"offset\":1140662,\"file\":{\"path\":\"/test.log\"}},\"message\":\"{\\\"log\\\":\\\"ttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest\\\\\\\",\\\\\\\"time\\\\\\\":\\\\\\\"2020-05-25T09:52:55Z\\\\\\\"}\\\\n\\\",\\\"stream\\\":\\\"stderr\\\",\\\"time\\\":\\\"2020-05-25T09:52:55.735481474Z\\\"}\",\"input\":{\"type\":\"log\"},\"host\":{\"name\":\"5d535347fb0a\"},\"agent\":{\"id\":\"579e046a-09ee-4cc8-91d2-4c82db4ef57e\",\"version\":\"7.6.0\",\"type\":\"filebeat\",\"ephemeral_id\":\"745c5775-ccc0-431d-a8d3-28299ce04ae9\",\"hostname\":\"5d535347fb0a\"},\"ecs\":{\"version\":\"1.4.0\"}}"
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 23, 2020, 2:42am UTC](https://discuss.elastic.co/t/filebeat-inptu-container-rotation-json-parser-error/233713/8 "2020-06-23T02:42:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.