# File beat monitor only recent log files
**URL:** https://discuss.elastic.co/t/file-beat-monitor-only-recent-log-files/52306
**Category:** Beats
**Tags:** filebeat
**Created:** [June 9, 2016, 9:25am UTC](https://discuss.elastic.co/t/file-beat-monitor-only-recent-log-files/52306 "2016-06-09T09:25:06Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@JohnnyS](https://discuss.elastic.co/u/JohnnyS)
#### Post date: [June 9, 2016, 9:25am UTC](https://discuss.elastic.co/t/file-beat-monitor-only-recent-log-files/52306/1 "2016-06-09T09:25:06Z")
I'm new with filebeat and want to know if it is possible to only monitor the log file of the day?
LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 6.7 (Santiago)
Release: 6.7
Filebeat config
paths:
- /opt/actional/profiles/ai\_st\_5400/logs/ActionalIntermediary\*.log
- /opt/actional/profiles/ai\_st\_5500/logs/ActionalIntermediary\*.log
- /opt/actional/profiles/aa\_st/logs/ActionalAgent\*.log
Content of one of the dirs. above is:
[actadm@el2027.bc]/opt/actional/profiles/ai\_st\_5400/logs # ls -al
drwxrwxr-x 2 actadm actadm 4096 Jun 7 00:00 ActionalIntermediary\_20160606\_000000\_023.ndx
-rw-rw-r-- 1 actadm actadm 34409511 Jun 7 23:59 ActionalIntermediary\_20160607\_000000\_024.log
drwxrwxr-x 2 actadm actadm 4096 Jun 8 00:00 ActionalIntermediary\_20160607\_000000\_024.ndx
-rw-rw-r-- 1 actadm actadm 34241535 Jun 8 23:59 ActionalIntermediary\_20160608\_000000\_025.log
drwxrwxr-x 2 actadm actadm 4096 Jun 9 00:00 ActionalIntermediary\_20160608\_000000\_025.ndx
-rw-rw-r-- 1 actadm actadm 16179493 Jun 9 11:22 ActionalIntermediary\_20160609\_000000\_026.log
drwxrwxr-x 2 actadm actadm 4096 Jun 9 00:00 ActionalIntermediary\_20160609\_000000\_026.ndx
As example, I only want to use the last file, in this case this is :ActionalIntermediary\_20160609\_000000\_026.log
This file will be created daily with a different date.
I thought of parameter close\_older or ignore\_older but this seems NOT to work. Can anyone help me out please? Thanks.
Kind regards
Johnny
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [June 9, 2016, 11:18am UTC](https://discuss.elastic.co/t/file-beat-monitor-only-recent-log-files/52306/2 "2016-06-09T11:18:47Z")
I assume the problem with ignore\_older is that in most cases it also reads the file from the day before because it is not 24h old yet? Or what exactly happened when you set it to 24h?
---
### Author:  [@JohnnyS](https://discuss.elastic.co/u/JohnnyS)
#### Post date: [June 13, 2016, 1:13pm UTC](https://discuss.elastic.co/t/file-beat-monitor-only-recent-log-files/52306/3 "2016-06-13T13:13:51Z")
Hi again
I checked this again and I uses the parameter in the config called "ignore\_older: 12h" and if I start-up filebeat for the first time he doesn't read files which are modified over 12 hours ago (which is what I should be)
The problem is, if filebeat is stopped and started again, it reads all the log files which are mentioned in the directory at that moment.
I believe the problem is somewhere with the .filebeat which is created when filebeat starts up.
First time I started filebeat this .filebeat (reg file I believe) is created for the first time and looks like:
[actadm@el2027.bc]/opt/actional/filebeat/filebeat-1.2.2-x86\_64 # more .filebeat
{"/opt/actional/profiles/ai\_st\_5400/logs/ActionalIntermediary\_20160607\_000000\_024.log":{"source":"/opt/actional/profiles/ai\_st\_5400/logs/ActionalIntermediary\_20160607\_000000\_024.log","FileStateOS":{"inode":130277,"device":64771}},"/opt/
actional/profiles/ai\_st\_5400/logs/ActionalIntermediary\_20160608\_000000\_025.log":{"source":"/opt/actional/profiles/ai\_st\_5400/logs/ActionalIntermediary\_20160608\_000000\_025.log","FileStateOS":{"inode":128516,"device":64771}},"/opt/actiona
l/profiles/ai\_st\_5400/logs/ActionalIntermediary\_20160609\_000000\_026.log":{"source":"/opt/actional/profiles/ai\_st\_5400/logs/ActionalIntermediary\_20160609\_000000\_026.log","FileStateOS":{"inode":128538,"device":64771}},"/opt/actional/profi
les/ai\_st\_5400/logs/ActionalIntermediary\_20160610\_000000\_027.log":{"source":"/opt/actional/profiles/ai\_st\_5400/logs/ActionalIntermediary\_20160610\_000000\_027.log","FileStateOS":{"inode":128517,"device":64771}},"/opt/actional/profiles/ai\_
st\_5400/logs/ActionalIntermediary\_20160611\_000000\_028.log":{"source":"/opt/actional/profiles/ai\_st\_5400/logs/ActionalIntermediary\_20160611\_000000\_028.log","FileStateOS":{"inode":130603,"device":64771}},"/opt/actional/profiles/ai\_st\_5400
/logs/ActionalIntermediary\_20160612\_000000\_029.log":{"source":"/opt/actional/profiles/ai\_st\_5400/logs/ActionalIntermediary\_20160612\_000000\_029.log","FileStateOS":{"inode":130276,"device":64771}},"/opt/actional/profiles/ai\_st\_5400/logs/A
ctionalIntermediary\_20160613\_000000\_030.log":{"source":"/opt/actional/profiles/ai\_st\_5400/logs/ActionalIntermediary\_20160613\_000000\_030.log","offset":21503900,"FileStateOS":{"inode":130288,"device":64771}}}
The second time filebeat starts it uses the same .filebeat (reg file) I believe and then he reads all the files mentioned in the dir
---
### Author:  [@Kyucheol\_Yeo](https://discuss.elastic.co/u/Kyucheol_Yeo)
#### Post date: [June 15, 2016, 3:34am UTC](https://discuss.elastic.co/t/file-beat-monitor-only-recent-log-files/52306/4 "2016-06-15T03:34:15Z")
I also have same issue.
Our system generates a log every an hour .
Filebeat config
paths:
- ${SERVER\_PATH}/server\*
ignore\_older:1h (or close\_older:20m)
scan\_frequency: 1m
ex.
server.2016-06-15\_08.log
server.2016-06-15\_09.log
server.2016-06-15\_10.log
the filebeate debug log is repeated "Not harvesting, file didn't change:"
but it was absolutly changed
And I also set the ignore\_older : 1h (also tried to set close\_older too)
When i monitored filebeat log, it seemed like new file is loaded well.
but it didn't read the new log what if the log file is modified.
After then i restarted the filebeat, the filebeat read the new generated log files.
please let me know how to solve this problem, thanks.
best regards
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [June 16, 2016, 8:15am UTC](https://discuss.elastic.co/t/file-beat-monitor-only-recent-log-files/52306/5 "2016-06-16T08:15:19Z")
@JohnnyS: Which version of filebeat are you using? I think the offset is missing in the state file in parts of the above files. Could it be that you use 1.2.2 (which has a bug here)?
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [June 16, 2016, 8:16am UTC](https://discuss.elastic.co/t/file-beat-monitor-only-recent-log-files/52306/6 "2016-06-16T08:16:51Z")
@Kyucheol_Yeo Which filebeat version are you using? Which OS are you using?
---
### Author:  [@JohnnyS](https://discuss.elastic.co/u/JohnnyS)
#### Post date: [June 16, 2016, 8:42am UTC](https://discuss.elastic.co/t/file-beat-monitor-only-recent-log-files/52306/7 "2016-06-16T08:42:47Z")
Indeed, I was using the filebeat version 1.2.2 and since yesterday I installed the new version 1.2.3 and the problem seems to be solved now. Thanks.
---
### Author:  [@Kyucheol\_Yeo](https://discuss.elastic.co/u/Kyucheol_Yeo)
#### Post date: [June 19, 2016, 6:16am UTC](https://discuss.elastic.co/t/file-beat-monitor-only-recent-log-files/52306/8 "2016-06-19T06:16:25Z")
os version is [CentOS release 6.4 (Final)]
filebeat version is [filebeat-1.2.3-x86\_64.rpm]
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [June 20, 2016, 10:20am UTC](https://discuss.elastic.co/t/file-beat-monitor-only-recent-log-files/52306/9 "2016-06-20T10:20:45Z")
@Kyucheol_Yeo All the above tests were made with 1.2.3. If yes, any chance to tests 5.0.0-alpha3 to see if the problem persists? I somehow get the feeling this could be `inode` reuse issue: [https://github.com/elastic/beats/issues/1341](https://github.com/elastic/beats/issues/1341) More details on how we plan to solve this issue here: [https://github.com/elastic/beats/issues/1600](https://github.com/elastic/beats/issues/1600)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 30, 2016, 9:25am UTC](https://discuss.elastic.co/t/file-beat-monitor-only-recent-log-files/52306/10 "2016-06-30T09:25:49Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.