# FileBeat Sends the same file content again and again
**URL:** https://discuss.elastic.co/t/filebeat-sends-the-same-file-content-again-and-again/47219
**Category:** Beats
**Tags:** filebeat
**Created:** [April 13, 2016, 9:07am UTC](https://discuss.elastic.co/t/filebeat-sends-the-same-file-content-again-and-again/47219 "2016-04-13T09:07:04Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Gaurav\_Harsola](https://discuss.elastic.co/u/Gaurav_Harsola)
#### Post date: [April 13, 2016, 9:07am UTC](https://discuss.elastic.co/t/filebeat-sends-the-same-file-content-again-and-again/47219/1 "2016-04-13T09:07:04Z")
Hi,
I have file beat agent running on one machine with conf as
```auto
filebeat:
prospectors:
-
paths:
- /var/logs/apache/*.log
input_type: log
registry_file: /var/lib/filebeat/registry
output:
logstash:
hosts: ["localhost:5044"]
shipper:
logging:
files:
rotateeverybytes: 10485760 # = 10MB
```
When i fire command to send the output to Elastisearch from log stash .My logstash.conf is like
```auto
input {
beats {
port => 5044
}
}
filter {
grok {
match => {
"message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:r
esponse:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}'
}
}
date {
match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
locale => en
}
geoip {
source => "clientip"
}
}
output {
stdout {
codec => plain {
charset => "ISO-8859-1"
}
}
elasticsearch {
hosts => "localhost:9200"
}
}
```
and then
```auto
./logsatsh -f logstash.conf
```
it send the o/p to Elastisearch but WHEN I ADD NEW FILE AGAIN IN /var/log/apache/ folder which i mentioned in filebeat.yml ,it resend the already sent data again to those index so how to stop that file to send its data again if its already sent
Plz help me
Thanks
gaurav
---
### Author:  [@rajkamalkool6](https://discuss.elastic.co/u/rajkamalkool6)
#### Post date: [April 13, 2016, 9:32am UTC](https://discuss.elastic.co/t/filebeat-sends-the-same-file-content-again-and-again/47219/2 "2016-04-13T09:32:28Z")
Include `ignore_older` property in your filebeat.yml config. For eg if you have mention `ignore_older: 5s` then filebeat wont pickup the file if the file is not modified for past 5s.
---
### Author:  [@tudor](https://discuss.elastic.co/u/tudor)
#### Post date: [April 13, 2016, 9:35am UTC](https://discuss.elastic.co/t/filebeat-sends-the-same-file-content-again-and-again/47219/3 "2016-04-13T09:35:21Z")
Not sure I understand how you are adding a new file. If the original file is just renamed, Filebeat should normally notice that and not send the file again. Can you give more details about the rotation strategy?
---
### Author:  [@Gaurav\_Harsola](https://discuss.elastic.co/u/Gaurav_Harsola)
#### Post date: [April 13, 2016, 9:45am UTC](https://discuss.elastic.co/t/filebeat-sends-the-same-file-content-again-and-again/47219/4 "2016-04-13T09:45:47Z")
I am adding new file by creating a file under same [folder.SO](http://folder.SO) that when filebeat see new file it sent out the content to ES
---
### Author:  [@tudor](https://discuss.elastic.co/u/tudor)
#### Post date: [April 13, 2016, 9:48am UTC](https://discuss.elastic.co/t/filebeat-sends-the-same-file-content-again-and-again/47219/5 "2016-04-13T09:48:08Z")
So you simply add a new file and the contents of the other files are sent again? That sounds quite strange. Can you come up with step by step instructions on how to reproduce it?
---
### Author:  [@Gaurav\_Harsola](https://discuss.elastic.co/u/Gaurav_Harsola)
#### Post date: [April 13, 2016, 9:56am UTC](https://discuss.elastic.co/t/filebeat-sends-the-same-file-content-again-and-again/47219/6 "2016-04-13T09:56:06Z")
If you see my filebeat.yml file above, there i mentioned /var/log/apaches/\*.log.
Let suppose initially under that folder there is no file and i started my filebeat and logstash command..
when i add new file let suppose 'apache.log' under /var/log/apaches/ it matches with my filebeat.yml condition so it send data to ES.
If again i create a new file 'test.log' under same fodler then it matche with /var/log/apache and send both earlier file (apache.log) and test.log to ES .
---
### Author:  [@Gaurav\_Harsola](https://discuss.elastic.co/u/Gaurav_Harsola)
#### Post date: [April 13, 2016, 10:12am UTC](https://discuss.elastic.co/t/filebeat-sends-the-same-file-content-again-and-again/47219/7 "2016-04-13T10:12:26Z")
can you explain a little bit more ..I think this might be the solution
---
### Author:  [@rajkamalkool6](https://discuss.elastic.co/u/rajkamalkool6)
#### Post date: [April 13, 2016, 10:53am UTC](https://discuss.elastic.co/t/filebeat-sends-the-same-file-content-again-and-again/47219/8 "2016-04-13T10:53:13Z")
Include ignore\_older property like below in your config file and mention the time in sec. For eg below I have mentioned 10s. So filebeat will read the file and wait for 10s and it will started to read the next file if there is nor modification in that file. It wont read again that file. Please refer the below link for detail description.
> filebeat:
> prospectors:
> -
> paths:
> - /var/logs/apache/\*.log
> input\_type: log
> ignore\_older: 10s
> registry\_file: /var/lib/filebeat/registry
> **[Configure inputs | Filebeat Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html)**
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 9:53pm UTC](https://discuss.elastic.co/t/filebeat-sends-the-same-file-content-again-and-again/47219/9 "2017-07-05T21:53:22Z")