# Filebeat - Sends all the messages on every new message written to log file

**URL:** <https://discuss.elastic.co/t/filebeat-sends-all-the-messages-on-every-new-message-written-to-log-file/171223>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 7, 2019, 4:52am UTC](https://discuss.elastic.co/t/filebeat-sends-all-the-messages-on-every-new-message-written-to-log-file/171223 "2019-03-07T04:52:35Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ash\_coder](https://avatars.discourse-cdn.com/v4/letter/a/53a042/32.png) [@ash\_coder](https://discuss.elastic.co/u/ash_coder)\
**Post date:** [March 7, 2019, 4:52am UTC](https://discuss.elastic.co/t/filebeat-sends-all-the-messages-on-every-new-message-written-to-log-file/171223/1 "2019-03-07T04:52:35Z")

</div>

Hi,

I am running the filebeat with below configuration. Filebeat always publishes all the messages from the log file whenever there is a new message written into it instead of reading the messages after the offset value. I see that offset is being written properly in registry file. Could some one help me to avoid publishing duplicate messages and read only new messages?

```
  filebeat.inputs:
  - type: log
  enabled: true
  paths:
  - /Users/xxx/loging/logs/*.log
  ignore_older: 24h

  output.file:
  path: "/tmp/filebeat_1"
  filename: filebeat

Registry:
[{"source":"/Users/xxx/loging/logs/1.log","offset":76,"timestamp":"2019-03-06T23:18:36.786319-05:00","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":8620380190,"device":16777220}}]

```

I cloned the code from the master

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [March 7, 2019, 3:09pm UTC](https://discuss.elastic.co/t/filebeat-sends-all-the-messages-on-every-new-message-written-to-log-file/171223/2 "2019-03-07T15:09:06Z")

</div>

> [@ash\_coder](#):
>
> I am running the filebeat with below configuration. Filebeat always publishes all the messages from the log file whenever there is a new message written into it instead of reading the messages after the offset value. I see that offset is being written properly in registry file. Could some one help me to avoid publishing duplicate messages a

Hello @ash_coder this is a strange behavior, the only reason why Filebeat would start a file from the beginning is because the inode of the file on disk change. I am assuming that you are currently developing with the master branch.

To verify my theory:

1. Stop filebeat.
2. Get the inode of the file using `ls -i myfile.log`
3. Start filebeat
4. When filebeat send all the events from the file.
5. Get the inode of the file using `ls -i myfile.log`

I think the inode has changed.

Are you editing the log file with an editor and adding logs to it manually? If this is the case, some editors on saving will actually create a new file on disk, every new file has a new inode. When this happens Filebeat will read the file completely. If you want to keep the file you need to append to the file like this `echo "hello" \>\> myfile.log"

You have to append to the file to keep the same inode like

---

<div class="post-metadata">

**Author:** ![ash\_coder](https://avatars.discourse-cdn.com/v4/letter/a/53a042/32.png) [@ash\_coder](https://discuss.elastic.co/u/ash_coder)\
**Post date:** [March 7, 2019, 6:46pm UTC](https://discuss.elastic.co/t/filebeat-sends-all-the-messages-on-every-new-message-written-to-log-file/171223/3 "2019-03-07T18:46:25Z")

</div>

> [@pierhugues](#):
>
> To verify my theory:
> 
> 1. Stop filebeat.
> 2. Get the inode of the file using `ls -i myfile.log`
> 3. Start filebeat
> 4. When filebeat send all the events from the file.
> 5. Get the inode of the file using `ls -i myfile.log`

Thank you very much @pierhugues. This was it. I was editing my file through vi and that was creating new inode.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 4, 2019, 6:46pm UTC](https://discuss.elastic.co/t/filebeat-sends-all-the-messages-on-every-new-message-written-to-log-file/171223/4 "2019-04-04T18:46:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.