# Duplicated logs from logstash after append logs

**URL:** <https://discuss.elastic.co/t/duplicated-logs-from-logstash-after-append-logs/39097>\
**Category:** Logstash\
**Created:** [January 13, 2016, 11:56am UTC](https://discuss.elastic.co/t/duplicated-logs-from-logstash-after-append-logs/39097 "2016-01-13T11:56:31Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![carmelom](https://avatars.discourse-cdn.com/v4/letter/c/e19adc/32.png) [@carmelom](https://discuss.elastic.co/u/carmelom)\
**Post date:** [January 13, 2016, 11:56am UTC](https://discuss.elastic.co/t/duplicated-logs-from-logstash-after-append-logs/39097/1 "2016-01-13T11:56:31Z")

</div>

When my java application is updating the logs, logstash is re-reading the file from the begning and kibana is displaing twice the same file.  
I tried with this configuration :  
`input { file { path => "/home/ubuntu/ELK/examples/LogsFromLog4j/*.log" type => "oldLogs" sincedb_path => "/home/ubuntu/ELK/logstash-2.1.1/sincedb_log4j/sincedb_log4j.db" } }`

and this :  
`input { beats { type => beats port => 5044 } }`  
but in both cases I have duplicated logs.

Some tips ? What is wrong ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 13, 2016, 12:23pm UTC](https://discuss.elastic.co/t/duplicated-logs-from-logstash-after-append-logs/39097/2 "2016-01-13T12:23:54Z")

</div>

It sounds like you're not really appending to the file but rather rewriting it. How, exactly, is the file being updated? It looks like you're using Log4j; which appender and how is it configured?

---

<div class="post-metadata">

**Author:** ![carmelom](https://avatars.discourse-cdn.com/v4/letter/c/e19adc/32.png) [@carmelom](https://discuss.elastic.co/u/carmelom)\
**Post date:** [January 13, 2016, 12:26pm UTC](https://discuss.elastic.co/t/duplicated-logs-from-logstash-after-append-logs/39097/3 "2016-01-13T12:26:02Z")

</div>

Hi,  
on my test I opened the file and then modify it appending logs.

Does Logstash is considering like a new file?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 13, 2016, 12:32pm UTC](https://discuss.elastic.co/t/duplicated-logs-from-logstash-after-append-logs/39097/4 "2016-01-13T12:32:03Z")

</div>

> on my test I opened the file and then modify it appending logs.

This doesn't answer the question "how, exactly, is the file updated". Details are crucial! What does the code look like.

> Does Logstash is considering like a new file?

Possibly. You can crank up Logstash's logging by starting it with `--verbose` to get more clues about what's happening.

---

<div class="post-metadata">

**Author:** ![carmelom](https://avatars.discourse-cdn.com/v4/letter/c/e19adc/32.png) [@carmelom](https://discuss.elastic.co/u/carmelom)\
**Post date:** [January 13, 2016, 1:54pm UTC](https://discuss.elastic.co/t/duplicated-logs-from-logstash-after-append-logs/39097/5 "2016-01-13T13:54:39Z")

</div>

"on my test I opened the file and then modify it appending logs. "  
Step 1 : mylogs.log -\> empty  
Step 2 : open file mylogs.log -\> add new log in the file -\> save file  
`<dialogue> <server>localhost</server> <duration>165</duration> <outgoing><qvs_message appstate="getAPPSTATE_0"></qvs_record></qvs_message></outgoing> </dialogue>`  
--\> mylogs.log = 1 Log  
logstash (or filebeats) is reading the file the I can verify it with kibana and I have 1 log  
Step 3 : open file mylogs.log -\> add new log in the file -\> save file  
`<dialogue> <server>localhost</server> <duration>165</duration> <outgoing><qvs_message appstate="getAPPSTATE_1"></qvs_record></qvs_message></outgoing> </dialogue>`  
--\> mylogs.log = 2 Log  
logstash (or filebeats) is reading the file the I can verify it with kibana and I have 3 logs but I **extected 2 logs**

I hope this is well detailed

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 13, 2016, 2:28pm UTC](https://discuss.elastic.co/t/duplicated-logs-from-logstash-after-append-logs/39097/6 "2016-01-13T14:28:06Z")

</div>

Opening the file in a text editor will create a new file with a new inode. Don't do that. Make sure you append to the same file, e.g. like this:

```
echo "test log message" >> logfile.log
```

---

<div class="post-metadata">

**Author:** ![carmelom](https://avatars.discourse-cdn.com/v4/letter/c/e19adc/32.png) [@carmelom](https://discuss.elastic.co/u/carmelom)\
**Post date:** [January 13, 2016, 4:26pm UTC](https://discuss.elastic.co/t/duplicated-logs-from-logstash-after-append-logs/39097/7 "2016-01-13T16:26:40Z")

</div>

Much better, thank you.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:15am UTC](https://discuss.elastic.co/t/duplicated-logs-from-logstash-after-append-logs/39097/8 "2017-07-06T05:15:51Z")

</div>