# How to prevent old log appending existing log in elasticsearch

**URL:** <https://discuss.elastic.co/t/how-to-prevent-old-log-appending-existing-log-in-elasticsearch/163656>\
**Category:** Logstash\
**Created:** [January 10, 2019, 5:10am UTC](https://discuss.elastic.co/t/how-to-prevent-old-log-appending-existing-log-in-elasticsearch/163656 "2019-01-10T05:10:20Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vinit\_Kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinit_kumar/32/74784_2.png) [@Vinit\_Kumar](https://discuss.elastic.co/u/Vinit_Kumar)\
**Post date:** [January 10, 2019, 5:10am UTC](https://discuss.elastic.co/t/how-to-prevent-old-log-appending-existing-log-in-elasticsearch/163656/1 "2019-01-10T05:10:20Z")

</div>

I am using filebeat to get logs from the remote server and shipping it to logstash so it's working fine. But when new logs being appending in the source log file so filebeat reads those logs from the beginning and send it to logstash and logstash appending all logs with older logs in the elasticsearch even though we already have those older logs in elasticsearch so here repetition is happening of logs.

So my question is how to ship only new added log into logstash. Once new lines of log append in log file so those new files should ship to logstash to elasticsearch.

Here is my filebeat.yml

```auto
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/user/Documents/ELK/locals/*.log

```

logstash input is logstash-input.conf

```auto
input {
  beats {
    port => 5044
  }
}

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 10, 2019, 6:49am UTC](https://discuss.elastic.co/t/how-to-prevent-old-log-appending-existing-log-in-elasticsearch/163656/2 "2019-01-10T06:49:13Z")

</div>

Exactly how is data appended to the log files? Is Filebeat reading from a local disk?

---

<div class="post-metadata">

**Author:** ![Vinit\_Kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinit_kumar/32/74784_2.png) [@Vinit\_Kumar](https://discuss.elastic.co/u/Vinit_Kumar)\
**Post date:** [January 10, 2019, 7:02am UTC](https://discuss.elastic.co/t/how-to-prevent-old-log-appending-existing-log-in-elasticsearch/163656/3 "2019-01-10T07:02:04Z")

</div>

Hi @Christian_Dahlqvist,

Thanks for reply. Filebeat is reading from local and remote server.

Appending data is like

`Example:`  
I have 4 lines in log file and all 4 line are read by filebeat and send it to logstash to elasticsearch. And in elasticsearch log count is 4.

Now if in my log file new 2 lines of log is created then filebeat read all log (old 4 lines + new 2 lines) and send it to logstash to elasticsearch. And in elasticsearch count become 10 but it should be 6.

And one more issue is filebeat isn't reading the last line of a log file. Any solution for last line also?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 10, 2019, 7:33am UTC](https://discuss.elastic.co/t/how-to-prevent-old-log-appending-existing-log-in-elasticsearch/163656/4 "2019-01-10T07:33:01Z")

</div>

It is [not recommended to read files over networked storage](https://www.elastic.co/guide/en/beats/filebeat/current/faq.html#filebeat-network-volumes) using Filebeat as this can cause problems. It is also important that all lines end with a newline (could be the reason the last line is not read correctly) and that data is truly appended to the file so the inode does not change (which will cause the file to be reread). Replacing a file with a new one with appended data will generally cause it all to be reread as it appears as a new file to Filebeat even though it has the same name as the old file.

This is why I was asking whether files were on local storage and exactly how they are appended to.

---

<div class="post-metadata">

**Author:** ![Vinit\_Kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinit_kumar/32/74784_2.png) [@Vinit\_Kumar](https://discuss.elastic.co/u/Vinit_Kumar)\
**Post date:** [January 10, 2019, 11:06am UTC](https://discuss.elastic.co/t/how-to-prevent-old-log-appending-existing-log-in-elasticsearch/163656/5 "2019-01-10T11:06:39Z")

</div>

@Christian_Dahlqvist `First` as of now my log files are on local storage. But I'm doing it for my remote logs, are on the cloud.

`Second` while running filebeat, If I'm adding anything new in the local log file then it will read new logs along with old logs and send them to logstash to elasticsearch. But I'm expecting to read only new logs because old logs are already rad and shipped to logstash to elasticsearch.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 10, 2019, 11:12am UTC](https://discuss.elastic.co/t/how-to-prevent-old-log-appending-existing-log-in-elasticsearch/163656/6 "2019-01-10T11:12:39Z")

</div>

Exactly how are you adding new data to the local file? If you use an editor to do this it will usually create a new file with a new inode behind the scenes. If you make sure you properly append data, e.g. `echo "New log line\n" >> test.txt` on Linux, it should not resend all data.

---

<div class="post-metadata">

**Author:** ![Vinit\_Kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinit_kumar/32/74784_2.png) [@Vinit\_Kumar](https://discuss.elastic.co/u/Vinit_Kumar)\
**Post date:** [January 10, 2019, 11:22am UTC](https://discuss.elastic.co/t/how-to-prevent-old-log-appending-existing-log-in-elasticsearch/163656/7 "2019-01-10T11:22:15Z")

</div>

Right now I'm adding data manually to local log file but on my remote log file server will add new file automatically. After adding \n to the local log file so filebeat is able to read all the lines.

My other problem is why filebeat is reading and sending old logs along with new log and sending it to logstash to elasticsearch?

I want new logs which are appends to my log file and want to ship them to logstash to elasticsearch.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 10, 2019, 11:24am UTC](https://discuss.elastic.co/t/how-to-prevent-old-log-appending-existing-log-in-elasticsearch/163656/8 "2019-01-10T11:24:31Z")

</div>

> [@Vinit\_Kumar](#):
>
> My other problem is why filebeat is reading and sending old logs along with new log and sending it to logstash to elasticsearch?

I explained this in my previous response. You need to append to the same file, which you can not do using an editor.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 7, 2019, 11:24am UTC](https://discuss.elastic.co/t/how-to-prevent-old-log-appending-existing-log-in-elasticsearch/163656/9 "2019-02-07T11:24:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.