Filebeat seems truncated message in single line[Filebeat 7.6]

Elastic Stack Beats
1

filebeat truncated log file line accidentally, so when parsing the next line, error happend

Filebeat error log

2020-04-28T16:50:37.642+0800 ERROR readjson/json.go:52 Error decoding JSON: invalid character 'e' in literal true (expecting 'r')
some correct line in files like below:

{"#account_id":"749626","#distinct_id":"5E63AE30-6EA8-4432-9FFE-70F63D444856","#type":"user_set","#ip":"112.96.193.209","#time":"2020-04-28 00:24:48","#event_name":"ROLE_CAPTURE","#account_no":"18890327","properties":{"role_uid":2320028591157306,"current_server":2103,"create_server":2103,"channel":"10013","role_name":"xxx","role_create_time":"2020-04-18 22:09:43","total_rmb":140,"role_level":57,"unionid_uid":4735535412207922585,"unionid_name":"faimily","ad":"","device":"5E63AE30-6EA8-4432-9FFE-70F63D444856","os":"2","os_version":"iPhone OS 9.3.3","role_type":0,"fighting":431671,"pay":true,"exp":39838,"level":57,"money":0,"gift_money":280,"gold":10496,"daohang":300,"contribution":105590,"achieve":795,"vigour":10,"country":"0","dungeon_ticket":3,"friend_point":350,"account_time":"2020-04-18 22:09:18","idfa":"","vitality":2,"vip_level":3,"ROLE_CAPTURE_total_login_day":10,"ROLE_CAPTURE_total_create_day":9,"ROLE_CAPTURE_last_login_time":"2020-04-28 00:24:48"}}

filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /data/datacenter/td_data/server_2101/*
  tags: ["doujiang-xiao7-2101"]
  fields:
    index: 'doujiang'
  harvester_buffer_size: 5242880
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
  reload.period: 30s
setup.template.name: "doujiang"
setup.template.pattern: "doujiang-*"
setup.template.fields: "doujiang.yml"
setup.template.overwrite: true
setup.kibana:
output.kafka:
  hosts: ["kafka1:9092", "kafka2:9092", "kafka3:9092"]
  topic: 'doujiang'
  reachable_only: false
  required_acks: 1
  compression: gzip
processors:
  - drop_fields:
      fields: ["offset", "beat.name","beat.version","beat.hostname","input.type", "ecs", "agent.ephemeral_id", "input"]
ignore_older: 2h
close_older: 2h
setup.ilm.enabled: false
queue.mem.events: 512
queue.mem.flush.min_events: 128
max_procs: 1
harvester_limit: 60000

OS: centos7.5 64bit
CPU: 4 core
MEM: 16Gb
2

Could you please share debug logs (./filebeat -e -d "*") possibly when Filebeat is encountering a line it truncates?

3

sorry, i have too many input files,output so many and so quickly, the speed is faster than the gif shows

filebeat
filebeat1144×488 1.4 MB

i did not always encounter this situation.
there is almost more or less 100 pieces from 100,000, which is not tolerated.
so, i have no idea how to capture the debug or error message for you.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.