Filebeat consuming disk space

ErSumit · November 26, 2018, 9:34am

Filebeat process is utilizing disk space. No clue why and where?
Whenever disk utilization reaches high, I restarted filebeat service and disk utilization reduced.
How can I fix this?

'''
[ec2-user@tomcat Filesystem /dev/xvda1 106G 101G 14M 100% / tmpfs 7.9G [ec2-user@tomcat 2018-11-26T09:06:23.060Z 2018-11-26T09:06:23.061Z 2018-11-26T09:06:23.061Z 2018-11-26T09:06:23.062Z 2018-11-26T09:06:23.062Z 2018-11-26T09:06:23.081Z 2018-11-26T09:06:23.084Z 2018-11-26T09:06:23.085Z 2018-11-26T09:06:23.086Z Config OK
Stopping filebeat: Starting filebeat: 2018-11-26T09:06:24.259Z 2018-11-26T09:06:24.260Z 2018-11-26T09:06:24.260Z 2018-11-26T09:06:24.260Z 2018-11-26T09:06:24.261Z ~] df -H Size Used Avail Use% Mounted on 0 7.9G 0% /dev/shm ~] sudo service filebeat restart
INFO instance/beat.go:492 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
INFO instance/beat.go:499 Beat UUID: xxxxxxxxxxxxxxxxxxxxx-xx-xx-x-xxxxxxxxxxxxxxxxxxxxx
INFO [beat] instance/beat.go:716 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "xxxxxxxxxxxxxxxxxxxxx"}}}
INFO [beat] instance/beat.go:725 Build info {"system_info": {"build": {"commit": "xxxxxxxxxxxxxxxxxxxxx", "libbeat": "6.3.2", "time": "2018-07-20T04:18:19.000Z", "version": "6.3.2"}}}
INFO [beat] instance/beat.go:728 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.9.4"}}}
INFO [beat] instance/beat.go:732 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-10-19T22:44:57+01:00","containerized":true,"hostname":"tomcat.xxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxx.com","ips":["127.0.0.1/8","::1/128","xx.x.xx.xx/24","xxx::xxx:xxx:xxx:xxx/64"],"kernel_version":"2.6.32-754.3.5.el6.x86_64","mac_addresses":["xx:xx:xx:xx:xx:xx"],"os":{"family":"redhat","platform":"redhat","name":"Red","version":"6.10 (Santiago)","major":6,"minor":10,"patch":0,"codename":"Santiago"},"timezone":"GMT","timezone_offset_sec":0}}}
INFO [beat] instance/beat.go:761 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63"],"ambient":null}, "cwd": "/", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 15150, "ppid": 15149, "seccomp": {"mode":""}, "start_time": "2018-11-26T09:06:22.000Z"}}}
INFO instance/beat.go:225 Setup Beat: filebeat; Version: 6.3.2
INFO pipeline/module.go:81 Beat name: i-xxxxxxxxx
[ OK ]
2018-11-26T09:06:24.259Z INFO instance/beat.go:492 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
INFO instance/beat.go:499 Beat UUID: xxx-xxxxxx-xxxxxx-xx-xxxx
INFO [beat] instance/beat.go:716 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "xxxxxxxxxxxxxxxxxxxxx"}}}
INFO [beat] instance/beat.go:725 Build info {"system_info": {"build": {"commit": "xxxxxxxxxxxxxxxxxxxxx", "libbeat": "6.3.2", "time": "2018-07-20T04:18:19.000Z", "version": "6.3.2"}}}
INFO [beat] instance/beat.go:728 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.9.4"}}}
INFO [beat] instance/beat.go:732 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-10-19T22:44:57+01:00","containerized":true,"hostname":"tomcat.xxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxx.com","ips":["127.0.0.1/8","::1/128","xx.x.xx.xx/24","xxxx:x:x:x/64"],"kernel_version":"2.6.32-754.3.5.el6.x86_64","mac_addresses":["x:x:x:x:x:x"],"os":{"family":"redhat","platform":"redhat","name":"Red","version":"6.10 (Santiago)","major":6,"minor":10,"patch":0,"codename":"Santiago"},"timezone":"GMT","timezone_offset_sec":0}}}

ErSumit · November 26, 2018, 9:35am

2018-11-26T09:06:24.261Z INFO [beat] instance/beat.go:761 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63"],"ambient":null}, "cwd": "/", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 15167, "ppid": 15166, "seccomp": {"mode":""}, "start_time": "2018-11-26T09:06:23.420Z"}}}
2018-11-26T09:06:24.261Z INFO instance/beat.go:225 Setup Beat: filebeat; Version: 6.3.2
2018-11-26T09:06:24.262Z INFO pipeline/module.go:81 Beat name: i-xxxxxxxxxxxx
Config OK
[ OK ]
[ec2-user@tomcat ~]$ df -H
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 106G 63G 38G 63% /
tmpfs 7.9G 0 7.9G 0% /dev/shm

[root@tomcat ec2-user]# ls -lrt /var/log/filebeat/
total 69404
-rw-------. 1 root root 10484957 Nov 7 00:49 filebeat.7
-rw-------. 1 root root 10485416 Nov 10 13:51 filebeat.6
-rw-------. 1 root root 10485157 Nov 14 02:49 filebeat.5
-rw-------. 1 root root 10485480 Nov 16 11:42 filebeat.4
-rw-------. 1 root root 10485264 Nov 19 19:35 filebeat.3
-rw-------. 1 root root 10307123 Nov 23 13:56 filebeat.2
-rw-------. 1 root root 8281145 Nov 26 09:06 filebeat.1
-rw-------. 1 root root 16780 Nov 26 09:06 filebeat
'''

ErSumit · November 26, 2018, 9:36am

I have not changed much of default parameters
Filebeat is serving about 20 files 24*7 to logstash server

Please help!

kvch · November 26, 2018, 10:21am

Is is possible that Filebeat has too many open file handles? What is the output of lsof -c filebeat?

ErSumit · November 26, 2018, 12:41pm

Yes, kvch
In output of 'lsof -c filebeat' I can see files from last 4 days.

How can I fix this? In my case files are named as --YYYY.mm.dd.log
and files only of todays date will get updated.

But I can see this line in logfile
log/harvester.go:253 File is inactive: --YYYY.mm.dd.log. Closing because close_inactive of 5m0s reached.
don't this closes file open handler?

ErSumit · November 26, 2018, 12:46pm

Even I can see lines from file of 23rd Nov is processed today. I can see them on kibana dashboard -> Discover

Log Message of 23rd Nov is arrived to logstash and thus elasticsearch today(26th Nov). Even this is causing me in wrong data.
What could be the reason? Please help in this as well.

I have 5 instances of filebeat and 5 logstash instances, as well.
The cluster consist of 3 master + 2 data nodes

system · December 24, 2018, 12:46pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.