FileBeat consumes free disk space until restart

Elastic Stack Beats
1

Hello!
Could you please help with Filebeat configuration.
Now we have a group of servers with the same Linux version (Ubuntu xenial) and the same FileBeat 5.5.1.
Zabbix sometimes handles empty disk space at one server from those group.
It happens not continuously and make some troubles until I restart filebeat service manually.
size of logs per hour is about 3.5Gb.

Filebeat configuration:

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/nginx/access.json.log
  exclude_lines: ["HEAD"]
  document_type: nginx-access-cdn-json

#----------------------------- Logstash output --------------------------------
output.logstash:
  hosts: ["example.com:5044"]
  compression_level: 3
  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

logging:
  level: info

LogRate config for nginx:

/var/log/nginx/*.log {
	maxsize 2G
        rotate 12
        hourly
        missingok
        compress
        delaycompress
        create 0640 www-data adm
        sharedscripts
        prerotate
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi \
        endscript
        postrotate
	        kill -USR1 $(cat /var/run/nginx.pid)
	        sleep 1
        endscript
}
2

What is the output of lsof -c filebeat? I assume your problem is that Filebeat does not let go of file descriptors.

You could add close_* options to your prospector configuration. This instructs Filebeat to close files e.g when they are deleted or renamed, etc. See more on these config options: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#filebeat-input-log-close-options

Let me know if you need further help.

3

Actually i tried option close_timeout = 2h, but it did not help me.
Thanks for lsof, I am monitoring it now (works properly without any close_* options) and when it happens again.
Anyway I will post a result of my investigation :slight_smile:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.