# Filebeat keeps sending same lines

**URL:** https://discuss.elastic.co/t/filebeat-keeps-sending-same-lines/83579
**Category:** Beats
**Tags:** filebeat
**Created:** [April 25, 2017, 2:03pm UTC](https://discuss.elastic.co/t/filebeat-keeps-sending-same-lines/83579 "2017-04-25T14:03:57Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![sLuvpreet33](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sluvpreet33/32/50626_2.png) [@sLuvpreet33](https://discuss.elastic.co/u/sLuvpreet33)
#### Post date: [April 25, 2017, 2:03pm UTC](https://discuss.elastic.co/t/filebeat-keeps-sending-same-lines/83579/1 "2017-04-25T14:03:57Z")

</div>

It is written that filebeat keeps the state of the file it reads in the registry file.  
So, it should not send the same logs again and again, if the log file has not changed.

I have added a cronjob which restarts filebeat every 5 minutes, so every 5 minutes, the same log data is being sent to the elastcisearch, whereas it should not send the same data again and again, am I correct ?

```
filebeat.registry_file: ${path.data}/registry

```

This line is in the `filebeat.full.yml`, do I have to add it in `filebeat.yml` also ?

---

<div class="post-metadata">

### Author: ![vinceh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinceh/32/17475_2.png) [@vinceh](https://discuss.elastic.co/u/vinceh)
#### Post date: [April 25, 2017, 3:18pm UTC](https://discuss.elastic.co/t/filebeat-keeps-sending-same-lines/83579/2 "2017-04-25T15:18:55Z")

</div>

filebeat.full.yml is an example file, it isn't read by filebeat by  
default.

You can pass -c to your invocation to specify it, or rename it to  
filebeat.yml.

---

<div class="post-metadata">

### Author: ![sLuvpreet33](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sluvpreet33/32/50626_2.png) [@sLuvpreet33](https://discuss.elastic.co/u/sLuvpreet33)
#### Post date: [April 25, 2017, 3:55pm UTC](https://discuss.elastic.co/t/filebeat-keeps-sending-same-lines/83579/3 "2017-04-25T15:55:54Z")

</div>

OR  
can I write this variable in filebeat.yml file ?

---

<div class="post-metadata">

### Author: ![vinceh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinceh/32/17475_2.png) [@vinceh](https://discuss.elastic.co/u/vinceh)
#### Post date: [April 25, 2017, 4:38pm UTC](https://discuss.elastic.co/t/filebeat-keeps-sending-same-lines/83579/4 "2017-04-25T16:38:39Z")

</div>

Yes.

filebeat.full.yml is a complete configuration example. Everything in it  
can be added to filebeat.yml.

---

<div class="post-metadata">

### Author: ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [April 26, 2017, 9:54am UTC](https://discuss.elastic.co/t/filebeat-keeps-sending-same-lines/83579/5 "2017-04-26T09:54:45Z")

</div>

> [@sLuvpreet33](#):
>
> It is written that filebeat keeps the state of the file it reads in the registry file.So, it should not send the same logs again and again, if the log file has not changed.

Filebeat keeps the most recent acknowledged state of files in the registry file. ACK is done by logstash/elasticsearch. As filebeat buffers lines into batches, lines read is \> lines ACKed. If output does not ACK an event, it must be send again (send-at-least-once-semantics).

Have you checked filebeat logs. Was connection closed while waiting for ACK?

Can you point me to the place in docs it's saying "read"? Might be a doc-bug.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 24, 2017, 10:04am UTC](https://discuss.elastic.co/t/filebeat-keeps-sending-same-lines/83579/6 "2017-05-24T10:04:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
