# Logstash sending the complete file when new data is added to the log file

**URL:** <https://discuss.elastic.co/t/logstash-sending-the-complete-file-when-new-data-is-added-to-the-log-file/55629>\
**Category:** Logstash\
**Created:** [July 15, 2016, 2:43pm UTC](https://discuss.elastic.co/t/logstash-sending-the-complete-file-when-new-data-is-added-to-the-log-file/55629 "2016-07-15T14:43:21Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sri\_ram](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@Sri\_ram](https://discuss.elastic.co/u/Sri_ram)\
**Post date:** [July 15, 2016, 2:43pm UTC](https://discuss.elastic.co/t/logstash-sending-the-complete-file-when-new-data-is-added-to-the-log-file/55629/1 "2016-07-15T14:43:21Z")

</div>

Hi,

Given the log file (apache.log) as input to Logstash, after applying some filter the output data is stored in Elasticsearch. Here, whenever the new data is added to the log file, logstash is sending the complete file to elasticsearch instead of updating the new events.

For example, for the first time apache.log file contains 10 lines of data, after parsing this file using logstash i have verified the count in elasticsearch. It's showed me **docs.count** is " **10**". After sometime, there are 5 new lines added to the apache.log file, logstash started sending events to elasticsearch. This time i have verified the count again, it's showing the **docs.count** is " **25**". I think the docs.count should be 15, not sure.

Any ideas/suggestions would be helpful.

The logstash configuration file is below:

```
input {
    file {
        path => "/path/to/logfile/"
        start_position => beginning
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
}
output {
    elasticsearch {}
    stdout {}
}
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 17, 2016, 8:33am UTC](https://discuss.elastic.co/t/logstash-sending-the-complete-file-when-new-data-is-added-to-the-log-file/55629/2 "2016-07-17T08:33:23Z")

</div>

> [@Sri\_ram](#):
>
> start\_position =\> beginning

That may be why.  
When the files is updated, it's not creating a new file with the same name is it?

---

<div class="post-metadata">

**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\
**Post date:** [July 17, 2016, 2:36pm UTC](https://discuss.elastic.co/t/logstash-sending-the-complete-file-when-new-data-is-added-to-the-log-file/55629/3 "2016-07-17T14:36:48Z")

</div>

Another possible reason is that 'file' input plugin doesn't manage to store sincedb file.  
Do you see a file named like ~/.sincedb ?  
If not, I advice you to explicitly set `sincedb_path => "/path/to/sincedb"`

---

<div class="post-metadata">

**Author:** ![Sri\_ram](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@Sri\_ram](https://discuss.elastic.co/u/Sri_ram)\
**Post date:** [July 17, 2016, 9:28pm UTC](https://discuss.elastic.co/t/logstash-sending-the-complete-file-when-new-data-is-added-to-the-log-file/55629/4 "2016-07-17T21:28:19Z")

</div>

The same log file is getting updated with new data. It's not creating another file with same name.

---

<div class="post-metadata">

**Author:** ![Sri\_ram](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@Sri\_ram](https://discuss.elastic.co/u/Sri_ram)\
**Post date:** [July 17, 2016, 9:32pm UTC](https://discuss.elastic.co/t/logstash-sending-the-complete-file-when-new-data-is-added-to-the-log-file/55629/5 "2016-07-17T21:32:46Z")

</div>

There is a since\_db file, which is created under home directory (~/.sincedb\_514abc). It has some values inside the file, for three columns.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 27, 2016, 4:40am UTC](https://discuss.elastic.co/t/logstash-sending-the-complete-file-when-new-data-is-added-to-the-log-file/55629/6 "2016-07-27T04:40:22Z")

</div>

@Sri_ram If you have resolved this, please put the solution in here. It may help someone in future 🙂

---

<div class="post-metadata">

**Author:** ![Sri\_ram](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@Sri\_ram](https://discuss.elastic.co/u/Sri_ram)\
**Post date:** [July 27, 2016, 1:48pm UTC](https://discuss.elastic.co/t/logstash-sending-the-complete-file-when-new-data-is-added-to-the-log-file/55629/7 "2016-07-27T13:48:57Z")

</div>

It was problem with vi editor. Whenever i tried to add data manually in to the file using vi editor, some editors creates new file instead of adding the data in to existing file. By using the command echo 'message' \>\> filename, i have added the data in to the log file. After that Logstash is sending only the updated events.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:46am UTC](https://discuss.elastic.co/t/logstash-sending-the-complete-file-when-new-data-is-added-to-the-log-file/55629/8 "2017-07-06T04:46:18Z")

</div>