Filebeat consuming disk space


(Sumit Sontakke) #1

Filebeat process is utilizing disk space. No clue why and where?
Whenever disk utilization reaches high, I restarted filebeat service and disk utilization reduced.
How can I fix this?

'''
[ec2-user@tomcat ~] df -H Filesystem Size Used Avail Use% Mounted on /dev/xvda1 106G 101G 14M 100% / tmpfs 7.9G 0 7.9G 0% /dev/shm [ec2-user@tomcat ~] sudo service filebeat restart
2018-11-26T09:06:23.060Z INFO instance/beat.go:492 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2018-11-26T09:06:23.061Z INFO instance/beat.go:499 Beat UUID: xxxxxxxxxxxxxxxxxxxxx-xx-xx-x-xxxxxxxxxxxxxxxxxxxxx
2018-11-26T09:06:23.061Z INFO [beat] instance/beat.go:716 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "xxxxxxxxxxxxxxxxxxxxx"}}}
2018-11-26T09:06:23.062Z INFO [beat] instance/beat.go:725 Build info {"system_info": {"build": {"commit": "xxxxxxxxxxxxxxxxxxxxx", "libbeat": "6.3.2", "time": "2018-07-20T04:18:19.000Z", "version": "6.3.2"}}}
2018-11-26T09:06:23.062Z INFO [beat] instance/beat.go:728 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.9.4"}}}
2018-11-26T09:06:23.081Z INFO [beat] instance/beat.go:732 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-10-19T22:44:57+01:00","containerized":true,"hostname":"tomcat.xxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxx.com","ips":["127.0.0.1/8","::1/128","xx.x.xx.xx/24","xxx::xxx:xxx:xxx:xxx/64"],"kernel_version":"2.6.32-754.3.5.el6.x86_64","mac_addresses":["xx:xx:xx:xx:xx:xx"],"os":{"family":"redhat","platform":"redhat","name":"Red","version":"6.10 (Santiago)","major":6,"minor":10,"patch":0,"codename":"Santiago"},"timezone":"GMT","timezone_offset_sec":0}}}
2018-11-26T09:06:23.084Z INFO [beat] instance/beat.go:761 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63"],"ambient":null}, "cwd": "/", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 15150, "ppid": 15149, "seccomp": {"mode":""}, "start_time": "2018-11-26T09:06:22.000Z"}}}
2018-11-26T09:06:23.085Z INFO instance/beat.go:225 Setup Beat: filebeat; Version: 6.3.2
2018-11-26T09:06:23.086Z INFO pipeline/module.go:81 Beat name: i-xxxxxxxxx
Config OK
Stopping filebeat: [ OK ]
Starting filebeat: 2018-11-26T09:06:24.259Z INFO instance/beat.go:492 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2018-11-26T09:06:24.259Z INFO instance/beat.go:499 Beat UUID: xxx-xxxxxx-xxxxxx-xx-xxxx
2018-11-26T09:06:24.260Z INFO [beat] instance/beat.go:716 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "xxxxxxxxxxxxxxxxxxxxx"}}}
2018-11-26T09:06:24.260Z INFO [beat] instance/beat.go:725 Build info {"system_info": {"build": {"commit": "xxxxxxxxxxxxxxxxxxxxx", "libbeat": "6.3.2", "time": "2018-07-20T04:18:19.000Z", "version": "6.3.2"}}}
2018-11-26T09:06:24.260Z INFO [beat] instance/beat.go:728 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.9.4"}}}
2018-11-26T09:06:24.261Z INFO [beat] instance/beat.go:732 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-10-19T22:44:57+01:00","containerized":true,"hostname":"tomcat.xxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxx.com","ips":["127.0.0.1/8","::1/128","xx.x.xx.xx/24","xxxx::x:x:x:x/64"],"kernel_version":"2.6.32-754.3.5.el6.x86_64","mac_addresses":["x:x:x:x:x:x"],"os":{"family":"redhat","platform":"redhat","name":"Red","version":"6.10 (Santiago)","major":6,"minor":10,"patch":0,"codename":"Santiago"},"timezone":"GMT","timezone_offset_sec":0}}}


(Sumit Sontakke) #2

2018-11-26T09:06:24.261Z INFO [beat] instance/beat.go:761 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63"],"ambient":null}, "cwd": "/", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 15167, "ppid": 15166, "seccomp": {"mode":""}, "start_time": "2018-11-26T09:06:23.420Z"}}}
2018-11-26T09:06:24.261Z INFO instance/beat.go:225 Setup Beat: filebeat; Version: 6.3.2
2018-11-26T09:06:24.262Z INFO pipeline/module.go:81 Beat name: i-xxxxxxxxxxxx
Config OK
[ OK ]
[ec2-user@tomcat ~]$ df -H
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 106G 63G 38G 63% /
tmpfs 7.9G 0 7.9G 0% /dev/shm

[root@tomcat ec2-user]# ls -lrt /var/log/filebeat/
total 69404
-rw-------. 1 root root 10484957 Nov 7 00:49 filebeat.7
-rw-------. 1 root root 10485416 Nov 10 13:51 filebeat.6
-rw-------. 1 root root 10485157 Nov 14 02:49 filebeat.5
-rw-------. 1 root root 10485480 Nov 16 11:42 filebeat.4
-rw-------. 1 root root 10485264 Nov 19 19:35 filebeat.3
-rw-------. 1 root root 10307123 Nov 23 13:56 filebeat.2
-rw-------. 1 root root 8281145 Nov 26 09:06 filebeat.1
-rw-------. 1 root root 16780 Nov 26 09:06 filebeat
'''


(Sumit Sontakke) #3

I have not changed much of default parameters
Filebeat is serving about 20 files 24*7 to logstash server

Please help!


(Noémi Ványi) #4

Is is possible that Filebeat has too many open file handles? What is the output of lsof -c filebeat?


(Sumit Sontakke) #5

Yes, kvch
In output of 'lsof -c filebeat' I can see files from last 4 days.

How can I fix this? In my case files are named as --YYYY.mm.dd.log
and files only of todays date will get updated.

But I can see this line in logfile
log/harvester.go:253 File is inactive: --YYYY.mm.dd.log. Closing because close_inactive of 5m0s reached.
don't this closes file open handler?


(Sumit Sontakke) #6

Even I can see lines from file of 23rd Nov is processed today. I can see them on kibana dashboard -> Discover

Log Message of 23rd Nov is arrived to logstash and thus elasticsearch today(26th Nov). Even this is causing me in wrong data.
What could be the reason? Please help in this as well.

I have 5 instances of filebeat and 5 logstash instances, as well.
The cluster consist of 3 master + 2 data nodes