Hi.
I have a problem with Filebeats 7.4.0.
It generates a very large syslog. If I don't stop the service, it would take up all the disk space.
Attached configuration (filebeat.yml):
What do you mean exactly by very large syslog? Do you mean that its logs are big? Or the events forwarded are too big? Or are you getting too many events?
Looking quickly at your config, I can see an issue with it. But I am not sure if it is going to resolve your problem, as you haven't provided much information. You need to remove or configure multiline correctly. If you want to use multiline settings, you need to put it under your input and provide some settings: https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html
Example config:
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/*.log
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
close_inactive: 5m
Thanks for answering. I mean the size of the syslog file when the filebeats service is running.
In the attached image you can see how it occupies more than 200GB.
If I stop the filebeats service, the growth of the syslog file is slow.
Could you please share the debug logs (./filebeat -e -d "*")?
Thank you. here it is. I had filebeat running for 30 seconds. has generated a 15mb file. very big, right?
uploaded file to google drive
I don't see any unusual or suspicious thing in your logs. Everything is normal.
But if you are not happy with the log file sizes of Filebeat, you can configure it. I suggest you set logging.files.rotateeverybytes to a number which fits your infrastructure (https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html#_logging_files_rotateeverybytes). You can also set logging.files.keepfiles to limit the number of rotated files (https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html#_logging_files_keepfiles). You can find further logging options on the same page I have linked above if you need more control.
The problem does not come from the filebeats log. The problem occurs when the filebeats service is active, the syslog increases disproportionately.
If the service is stopped, syslog increases very little.
The links you provide correspond to the filebeat log, right?
Yes, it corresponds to the Filebeat log.
So just to make sure I understand you issue correctly, the problem is not with the amout of log written by Filebeat, right? Is there some service which depends on it and logs too much?
Right.
but it only happens when the filebeat service is running.
Can you share a few logs which are generated when Filebeat is running?
ok, first I will enable filebeat to reproduce the issue
attached.
any idea?
You´re watching /var/log/syslog and filebeat is logging to /var/log/syslog? Maybe you´ve created a loop?
Try to disable watching of /var/log/syslog. And check if it´s getting better.
I took a look at the syslogfile you´ve attached.
The file ist 206MB. After deleting all Event with ""Cannot index event" the file is only 96MB. After filtering "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];. The file is only 70MB.
I´ll try to go deeper.
It seem that filebeat logs SQL-statements (coming from another logfile) to you syslog. I would guess that this is the most of the 70MB.
Why is filebeat writing in / var / log / syslog? I don't see any settings to write about this file.
This problem is only generated when the filebeat service is running. Right now with the service stopped, syslog has a size of 322Kb
Try to set the loglevel to "info":
https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html
You also have to fix the
"blocked by: [FORBIDDEN/12/index read-only / allow delete
Try this: https://stackoverflow.com/questions/50609417/elasticsearch-error-cluster-block-exception-forbidden-12-index-read-only-all
Or you configure filebeat to log to a different file. That can also be found here:
https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html
filebeat is logging by default in /var/log/filebeat/filebeat.log
I continue reading the links you have given me