Strange encoding issue each day. Need help to analyze it

Elastic Stack Beats
1

Hi all,

we are shipping logs using following route:
logfile <- filebeat -> redis <- logstash -> elasticsearch <- kibana

Using filebeat 6.5.2 on windows 2012 R1. Filebeat is installed as service.

Each day at about 23:00 local time we are facing a strange effect when we discover the data in kibana:
image
Then we have a block where all lines are looking that cryptic.
Lines before that block are fine.

image
Here you can see the garbage block. Much more lines than usual.

If i check the input logfile, all seems to be fine.

Other logfiles are not affected.

I checked if there are correctly parsed lines missing in kibana, e.g. to track down if some lines in the source logfile are causing this issue.
Looks like as if there is NO line missing. lasst message displayed ok before issue and first event without issue are next to each other in logfile.

Any Ideas? I will ask in datacenter if they are running a backup at this time, but it is strange that other logfiles are not effected.

2

I found the root cause: our daily log compression causes it. Filebeat is shipping the zip files, which are stored in the same folder.
So I need to reconfigure filebeat to ignore zip files.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.