# Logstash file Input Plugin to read all files from a directory where log files gets rotated every hour
**URL:** https://discuss.elastic.co/t/logstash-file-input-plugin-to-read-all-files-from-a-directory-where-log-files-gets-rotated-every-hour/270806
**Category:** Logstash
**Created:** [April 21, 2021, 9:03am UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-to-read-all-files-from-a-directory-where-log-files-gets-rotated-every-hour/270806 "2021-04-21T09:03:14Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@shadu88](https://discuss.elastic.co/u/shadu88)
#### Post date: [April 21, 2021, 9:03am UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-to-read-all-files-from-a-directory-where-log-files-gets-rotated-every-hour/270806/1 "2021-04-21T09:03:14Z")
Hello Dear ELkan's
Hope all are doing good!!.
here are details of my query:
1. Logstash version7.10 on Centos7.8
2. Using File input plugin to read files and forward logs to other components
3. Issue Statement : How can i read all file from a directory where .log files are getting rotated every hour without duplicating the data and loose any logs.
4. File pattern based on time eg. mainlog.log-2021042000
5. my configurations:
input {
file {
path =\> "/data/mypath/tologs/mainlog.log"
start\_position =\> "beginning"
sincedb\_path =\> "/etc/logstash/sincedb/null"
}
}
6. every hour new file is created
mainlog.log-2021042000
mainlog.log-2021042001
mainlog.log-2021042002
mainlog.log-2021042003
mainlog.log-2021042004
mainlog.log-2021042005
mainlog.log-2021042006
mainlog.log-2021042007
mainlog.log-2021042008
mainlog.log-2021042009
mainlog.log-2021042010
mainlog.log-2021042011
mainlog.log-2021042012
mainlog.log-2021042013
mainlog.log-2021042014
mainlog.log-2021042015
mainlog.log-2021042016
mainlog.log-2021042017
mainlog.log-2021042018
mainlog.log-2021042019
mainlog.log-2021042020
mainlog.log-2021042021
mainlog.log-2021042022
mainlog.log-2021042023
I appreciate your inputs on this thank you in advance.
Stay Safe (y)
---
### Author:  [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)
#### Post date: [April 21, 2021, 9:57am UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-to-read-all-files-from-a-directory-where-log-files-gets-rotated-every-hour/270806/2 "2021-04-21T09:57:11Z")
If you did `path => "/data/mypath/tologs/mainlog.log-*"` it will read all the files and there should be no duplicates as long as the files don't contain duplicates.
This part is telling it to start over from the beginning each time though. So once you get it working I would remove those and Logstash will maintain which logs and which data has been already ingested so it won't grab those again.
```auto
start_position => "beginning"
sincedb_path => "/etc/logstash/sincedb/null"
```
---
### Author:  [@shadu88](https://discuss.elastic.co/u/shadu88)
#### Post date: [April 22, 2021, 6:10am UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-to-read-all-files-from-a-directory-where-log-files-gets-rotated-every-hour/270806/3 "2021-04-22T06:10:36Z")
Hello Aaron,
Thanks for quick turnaround.
Absolutely path =\> "/data/mypath/tologs/mainlog.log-\*" will include all my log files, but the catch is same logs gets rotated every hour obviously its a duplicate data.
What's your recommendation for my scenario, where log files rotate every one hour.
Thanks Much!!!
Regards,
Shadab
---
### Author:  [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)
#### Post date: [April 22, 2021, 8:41am UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-to-read-all-files-from-a-directory-where-log-files-gets-rotated-every-hour/270806/4 "2021-04-22T08:41:35Z")
I am not sure if there is a way if your logs are creating duplicate data.
---
### Author:  [@shadu88](https://discuss.elastic.co/u/shadu88)
#### Post date: [April 22, 2021, 11:40am UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-to-read-all-files-from-a-directory-where-log-files-gets-rotated-every-hour/270806/5 "2021-04-22T11:40:45Z")
Currently my configuration is pointing to active file where logging happens actively.
path =\> "/data/mypath/tologs/mainlog.log" ( im loosing data by this configuration)
while file rotates to next timestamp file
every hour new file is created
mainlog.log-2021042000
mainlog.log-2021042001
mainlog.log-2021042002 etc etc.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 20, 2021, 11:41am UTC](https://discuss.elastic.co/t/logstash-file-input-plugin-to-read-all-files-from-a-directory-where-log-files-gets-rotated-every-hour/270806/6 "2021-05-20T11:41:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.