Problem with big log files

Elastic Stack Beats
1

Hi to all!
I am using filebeat version 6.8.8 and i have a problem. I need to bring deploy logs to kibana. Each file have 1-2k lines and for parsing in Logstash i need whole file content and there is my problem.
If i use config like this my logs come not completely. For example i have file 2k lines, first doc comes in size of 500 lines, that is, the same file comes in parts and that fact give me a lot of problem with message parsing in logstash, because message is not full

image
image850×344 29.7 KB

If i don't use :

  clean_inactive: 24h
  ignore_older: 23h
  close_inactive: 22h
  close_eof: true

then my registry file just overflows and i recieve ERROR every second:

ERROR	registrar/registrar.go:363	Writing of registry returned error: rename /var/lib/filebeat/registry.new /var/lib/filebeat/registry: no such file or directory. Continuing...

I really hope u understood my problem, I would be grateful if you help)

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.