Filebeat keeps open files forever 6.1.0 & Mesos


(이지홍) #1

I'm currently using Filebeat 6.1.0 with Arura&Mesos for collecting Tomcat logs Which are rotated every hour, and max history is six.
But sometimes, rotated log files are still using disk space because the Filebeat using them forever.
I checked using the lsof command.

abc.log
abc.log.yyyymmdd01(deleted)
abc.log.yyyymmdd02(deleted)
1513350959688

Here is my configuration of Filebeat
FileBeat:
prospectors:
-
paths:
- {{profile.logDesc.paths}}
fields:
type: {{profile.logDesc.doc_type}}
encoding: utf-8
ignore_older: 6m
close_inactive: 5m
close_removed: true
scan_frequency: 10s
clean_inactive: 10m
document_type: {{profile.logDesc.doc_type}}
registry_file: /home/irteamsu/.filebeat/.filebeat-apilog

I added some configuration to Filebeat config file, but it does not work intended.
(ignored_older/close_inactive/closed_removed/clean_inactive )
It happened several times in 10 times.
Do I need to set close_renamed: true too?

Please, somebody, teach me what is the problems and what should I do?

Best Regards
Jeehong Lee.


(Gostar) #2

You can try “close_timeout”

https://www.elastic.co/guide/en/beats/filebeat/current/faq.html#faq-deleted-files-are-not-freed


(이지홍) #3

Thank you very much
I will try it


(이지홍) #4

I found something strange things
Here are two files
first one is currently writing the file by Tomcat
the second one is already rotated a few min ago by Tomcat
the Inode of two files are different but, the timestamp(maybe updated time?) is same.
Does it ok?
I think the reason for above issue is this.

1575164 -rw-r--r-- 1 www www 11735119958 Dec 18 16:59 x_access_log.2017121816

1575219 -rw-r--r-- 1 www www 6880126694 Dec 18 17:31 x_access_log

{
"source": "/home/www/logs/tomcat/x_access_log",
"offset": 2158855741,
"timestamp": "2017-12-18T17:15:55.751004229+09:00",
"ttl": -1,
"type": "log",
"FileStateOS": {
"inode": 1575164,
"device": 2051
}
}, {
"source": "/home/www/logs/tomcat/x_access_log",
"offset": 320312960,
"timestamp": "2017-12-18T17:15:55.750996978+09:00",
"ttl": -1,
"type": "log",
"FileStateOS": {
"inode": 1575219,
"device": 2051
}
}


(ruflin) #5

The above looks normal to me. One of the events is from the file which was just rotated and the other one from the new file.

Normally files are kept open, because not all events were sent. Can you share your filebeat log file?

Also when pasting configs, please make sure the indentations are kept by adding ticks around the code par.


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.