Filebeat logs are too verbose

[karlsson]

Hi all. Is there a way to reduce the size of filebeat logs? I deployed filebeat to a Docker host, everything works as expected, all container logs are sent to Elasticsearch. However, after a few days, json log file in /var/lib/docker/containers was smth like 20GB. It was filebeat log file, other logs files were much smaller. Is it expected? Do I need to somehow monitor filebeat docker log?

[kvch (Noémi Ványi)]

I am not sure the root of your problem is Filebeat unless you have modified the default logging configuration. By default the maximum size of a log file (produced by Filebeat) is 10 MB. If the size exceeds it, the file gets rotated. The number of log files created by Filebeat is 7. Thus, total size of the logs should not be bigger than 70 MB.

You can configure logging of Filebeat in a few ways to avoid the issue you are having at the moment.

1. Decrease logging level
   You can set the logging level to warning or error, so the number of log messages are decreased. It can be adjusted using the option logging.level: https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html#level
2. Decrease the size of the log files or the number of rotated files
   Set a smaller size for the log by configuring logging.files.rotateeverybytes: https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html#_logging_files_rotateeverybytes Or you can lower the number of log files by setting logging.files.keepfiles: https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html#_logging_files_permissions