# How to use filebeat to collect only the day's logs **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [April 7, 2021, 6:49am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-to-collect-only-the-days-logs/269409 "2021-04-07T06:49:55Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [April 7, 2021, 6:49am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-to-collect-only-the-days-logs/269409/1 "2021-04-07T06:49:55Z")
I'm using filebeat to collect logs. The name of the target log is "api-yyyy-mm-dd.log". If you set the paths setting of filebeat as follows, all files will be collected. ```auto - type: log paths: - /var/log/api/api-*.log ``` How can I configure it to only collect files from the same day? ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [April 7, 2021, 9:54pm UTC](https://discuss.elastic.co/t/how-to-use-filebeat-to-collect-only-the-days-logs/269409/2 "2021-04-07T21:54:33Z")
I don't think there is a way to do that but it sounds like the application will only write to one file per day. Is that right? If so once filebeat completes reading the file, it won't read it anymore and will move on. Are you having an issue with the current setup? ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [April 8, 2021, 1:02am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-to-collect-only-the-days-logs/269409/3 "2021-04-08T01:02:11Z")
Thank you for your comment. The problem often arises during implementation. As you point out, the target log creates one file per day. There are two months of files with the creation date as the file name. Once the file is read, it is not read, but on the first day of installing filebeat, it tries to import all the logs in the folder. I thought it would be nice to be able to collect some scrutiny during input. ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [April 8, 2021, 1:15am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-to-collect-only-the-days-logs/269409/4 "2021-04-08T01:15:57Z")
Gotcha, ur trying to not import the old log files and only import everything from the day u install going forward? Can u not rename the old files to something so it doesn't match the pattern? ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [April 8, 2021, 1:21am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-to-collect-only-the-days-logs/269409/5 "2021-04-08T01:21:28Z")
Yeah. Do you think it would be smart to do so? If we make app.log for the day, and app-yyyy-mm-dd.log for the rotation, will that accomplish the job of logstash? ```auto - type: log paths: - /var/log/api/api.log ``` ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [April 8, 2021, 1:25am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-to-collect-only-the-days-logs/269409/6 "2021-04-08T01:25:32Z")
Now we deal with it this way at the beginning of the month. We think it is not very smart. ```auto - type: log paths: - /var/log/api/api-2021-04-*.log ``` ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [April 8, 2021, 1:34am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-to-collect-only-the-days-logs/269409/7 "2021-04-08T01:34:13Z")
I guess u could do that, but the registry should track everything so even if u reinstall or restart filebeat, as long as the registry is persistent it should reread it ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [April 8, 2021, 2:26am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-to-collect-only-the-days-logs/269409/8 "2021-04-08T02:26:41Z")
Can't I use the exclude\_files option? It seems to be used to exclude file extensions. ```auto exclude_files: [".gz$"]. ``` However, the following had no effect ```auto exclude_files: ["api-2021-02-*.log", "api-2021-03-*.log"]. ``` ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [April 8, 2021, 2:50am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-to-collect-only-the-days-logs/269409/9 "2021-04-08T02:50:35Z")
Is it possible that "\*" can be used for paths settings, but not for exclude\_files? It will work if you set it as follows ```auto exclude_files: ["api-2021-(02|03)-(0[1-9]|[1-2][0-9]|3[0-1]).log"] ``` ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [April 8, 2021, 3:35am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-to-collect-only-the-days-logs/269409/10 "2021-04-08T03:35:46Z")
Idk if the exclude files is using globs or regex... to do it's filtering, I'd have to look at the documentation. But that still doesn't seem manageable in the long term. If nothing else works, I think the idea u had about having the file rotation change the name would probably be best long-term since it's automated. ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [April 9, 2021, 2:02am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-to-collect-only-the-days-logs/269409/11 "2021-04-09T02:02:24Z")
> I think the idea u had about having the file rotation change the name would probably be best long-term since it's automated. I have a feeling your idea is correct. I will try to adjust the file name to see if I can change it. ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [April 9, 2021, 12:26pm UTC](https://discuss.elastic.co/t/how-to-use-filebeat-to-collect-only-the-days-logs/269409/12 "2021-04-09T12:26:00Z")
THe only issue with the log name rotation is if Filebeat is still reading the file when it rotates it can cause issues. It was in another thread. ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [April 12, 2021, 12:53am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-to-collect-only-the-days-logs/269409/13 "2021-04-12T00:53:16Z")
That's bad news for me. Given the nature of filebeat, I guess I shouldn't manipulate files while they are being read. ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [April 12, 2021, 1:05am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-to-collect-only-the-days-logs/269409/14 "2021-04-12T01:05:36Z")
If u use the default naming convention where the file names don't change and u get a new file each day, the registry should prevent previously read files from being reread even on restart and reinstall (provided the registry persists reinstall/update). ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [April 13, 2021, 1:05am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-to-collect-only-the-days-logs/269409/15 "2021-04-13T01:05:20Z")
I see. I was not aware of the registry. Can you tell me if this is a filebeat degree setting? ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [April 13, 2021, 3:04am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-to-collect-only-the-days-logs/269409/16 "2021-04-13T03:04:27Z")
See [How to understand "registry" file in filebeat - #2 by pierhugues](https://discuss.elastic.co/t/how-to-understand-registry-file-in-filebeat/157271/2) ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [April 14, 2021, 10:37am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-to-collect-only-the-days-logs/269409/17 "2021-04-14T10:37:21Z")
Thanks for adding the explanation of the registry. I don't think there is any need to rewrite `offset` in cases where the number of files to be read is newly increased (i.e. logs in the format yyyy-mm-dd.log, etc.), because the same file name will not change the reading position. In the case where the file name is changed due to log rotation (i.e., the log for the day is today.log, and the next day it is renamed to yyyy-mm-dd.log, etc.), it is not necessary to change the reading position of today.log. In this case, you need to change the reading position, so you need to change the `offset` and `FileStateOS`. In this case, you need to change the `offset` and `FileStateOS`. However, I don't know how to set these. In many operations, it only says to delete the registry if you want to re-read a log that has already been read. The only thing it says is that I think it is a common case to rename files for log rotation, etc. What are the common means of doing this? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 12, 2021, 12:37pm UTC](https://discuss.elastic.co/t/how-to-use-filebeat-to-collect-only-the-days-logs/269409/18 "2021-05-12T12:37:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.