File beat monitor only recent log files

[JohnnyS (Johnny Soetaerts)]

I'm new with filebeat and want to know if it is possible to only monitor the log file of the day?

LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 6.7 (Santiago)
Release: 6.7

Filebeat config

paths:
- /opt/actional/profiles/ai_st_5400/logs/ActionalIntermediary*.log
- /opt/actional/profiles/ai_st_5500/logs/ActionalIntermediary*.log
- /opt/actional/profiles/aa_st/logs/ActionalAgent*.log

Content of one of the dirs. above is:

[actadm@el2027.bc]/opt/actional/profiles/ai_st_5400/logs # ls -al
drwxrwxr-x 2 actadm actadm 4096 Jun 7 00:00 ActionalIntermediary_20160606_000000_023.ndx
-rw-rw-r-- 1 actadm actadm 34409511 Jun 7 23:59 ActionalIntermediary_20160607_000000_024.log
drwxrwxr-x 2 actadm actadm 4096 Jun 8 00:00 ActionalIntermediary_20160607_000000_024.ndx
-rw-rw-r-- 1 actadm actadm 34241535 Jun 8 23:59 ActionalIntermediary_20160608_000000_025.log
drwxrwxr-x 2 actadm actadm 4096 Jun 9 00:00 ActionalIntermediary_20160608_000000_025.ndx
-rw-rw-r-- 1 actadm actadm 16179493 Jun 9 11:22 ActionalIntermediary_20160609_000000_026.log
drwxrwxr-x 2 actadm actadm 4096 Jun 9 00:00 ActionalIntermediary_20160609_000000_026.ndx

As example, I only want to use the last file, in this case this is :ActionalIntermediary_20160609_000000_026.log

This file will be created daily with a different date.

I thought of parameter close_older or ignore_older but this seems NOT to work. Can anyone help me out please? Thanks.

Kind regards

Johnny

[ruflin]

I assume the problem with ignore_older is that in most cases it also reads the file from the day before because it is not 24h old yet? Or what exactly happened when you set it to 24h?

[JohnnyS (Johnny Soetaerts)]

Hi again

I checked this again and I uses the parameter in the config called "ignore_older: 12h" and if I start-up filebeat for the first time he doesn't read files which are modified over 12 hours ago (which is what I should be)

The problem is, if filebeat is stopped and started again, it reads all the log files which are mentioned in the directory at that moment.

I believe the problem is somewhere with the .filebeat which is created when filebeat starts up.

First time I started filebeat this .filebeat (reg file I believe) is created for the first time and looks like:

[actadm@el2027.bc]/opt/actional/filebeat/filebeat-1.2.2-x86_64 # more .filebeat
{"/opt/actional/profiles/ai_st_5400/logs/ActionalIntermediary_20160607_000000_024.log":{"source":"/opt/actional/profiles/ai_st_5400/logs/ActionalIntermediary_20160607_000000_024.log","FileStateOS":{"inode":130277,"device":64771}},"/opt/
actional/profiles/ai_st_5400/logs/ActionalIntermediary_20160608_000000_025.log":{"source":"/opt/actional/profiles/ai_st_5400/logs/ActionalIntermediary_20160608_000000_025.log","FileStateOS":{"inode":128516,"device":64771}},"/opt/actiona
l/profiles/ai_st_5400/logs/ActionalIntermediary_20160609_000000_026.log":{"source":"/opt/actional/profiles/ai_st_5400/logs/ActionalIntermediary_20160609_000000_026.log","FileStateOS":{"inode":128538,"device":64771}},"/opt/actional/profi
les/ai_st_5400/logs/ActionalIntermediary_20160610_000000_027.log":{"source":"/opt/actional/profiles/ai_st_5400/logs/ActionalIntermediary_20160610_000000_027.log","FileStateOS":{"inode":128517,"device":64771}},"/opt/actional/profiles/ai_
st_5400/logs/ActionalIntermediary_20160611_000000_028.log":{"source":"/opt/actional/profiles/ai_st_5400/logs/ActionalIntermediary_20160611_000000_028.log","FileStateOS":{"inode":130603,"device":64771}},"/opt/actional/profiles/ai_st_5400
/logs/ActionalIntermediary_20160612_000000_029.log":{"source":"/opt/actional/profiles/ai_st_5400/logs/ActionalIntermediary_20160612_000000_029.log","FileStateOS":{"inode":130276,"device":64771}},"/opt/actional/profiles/ai_st_5400/logs/A
ctionalIntermediary_20160613_000000_030.log":{"source":"/opt/actional/profiles/ai_st_5400/logs/ActionalIntermediary_20160613_000000_030.log","offset":21503900,"FileStateOS":{"inode":130288,"device":64771}}}

The second time filebeat starts it uses the same .filebeat (reg file) I believe and then he reads all the files mentioned in the dir

[Kyucheol_Yeo]

I also have same issue.
Our system generates a log every an hour .

Filebeat config
paths:

• ${SERVER_PATH}/server*
  ignore_older:1h (or close_older:20m)
  scan_frequency: 1m

ex.
server.2016-06-15_08.log
server.2016-06-15_09.log
server.2016-06-15_10.log

the filebeate debug log is repeated "Not harvesting, file didn't change:"
but it was absolutly changed

And I also set the ignore_older : 1h (also tried to set close_older too)

When i monitored filebeat log, it seemed like new file is loaded well.
but it didn't read the new log what if the log file is modified.

After then i restarted the filebeat, the filebeat read the new generated log files.

please let me know how to solve this problem, thanks.

best regards

[ruflin]

@JohnnyS: Which version of filebeat are you using? I think the offset is missing in the state file in parts of the above files. Could it be that you use 1.2.2 (which has a bug here)?

[ruflin]

@Kyucheol_Yeo Which filebeat version are you using? Which OS are you using?

[JohnnyS (Johnny Soetaerts)]

Indeed, I was using the filebeat version 1.2.2 and since yesterday I installed the new version 1.2.3 and the problem seems to be solved now. Thanks.

[Kyucheol_Yeo]

os version is [CentOS release 6.4 (Final)]
filebeat version is [filebeat-1.2.3-x86_64.rpm]

[ruflin]

@Kyucheol_Yeo All the above tests were made with 1.2.3. If yes, any chance to tests 5.0.0-alpha3 to see if the problem persists? I somehow get the feeling this could be inode reuse issue: https://github.com/elastic/beats/issues/1341 More details on how we plan to solve this issue here: https://github.com/elastic/beats/issues/1600