# Doubts in Filebeat

**URL:** <https://discuss.elastic.co/t/doubts-in-filebeat/55079>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [July 8, 2016, 6:12pm UTC](https://discuss.elastic.co/t/doubts-in-filebeat/55079 "2016-07-08T18:12:36Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pedro\_Lopes](https://avatars.discourse-cdn.com/v4/letter/p/e0b2c6/32.png) [@Pedro\_Lopes](https://discuss.elastic.co/u/Pedro_Lopes)\
**Post date:** [July 8, 2016, 6:12pm UTC](https://discuss.elastic.co/t/doubts-in-filebeat/55079/1 "2016-07-08T18:12:36Z")

</div>

Hello guys,

I have a filebeat.yml that works perfectly sending csv files to logstash, and then logstash to elastic.

If I add a new file to any of the folders filebeats starts it's magic. But if I add data to the end of a existing/read file, filebeats reads the hole file from start including new line. What I'm looking for is that filebeat reads the new added data for existing files. Is this possible?

Here is my filebeat.yml:

```auto
filebeat:
  prospectors:
    -
      paths:
        - "/home/vagrant/files/type_one*.csv"
      document_type: type_one
      ignore_older: 1m
    -
      paths:
        - "/home/vagrant/files/type_two*.csv"
      document_type: type_two
      ignore_older: 1m
    -
      paths:
        - "/home/vagrant/files/type_three*.csv"
      document_type: type_three
      ignore_older: 1m
output:
  logstash:
    hosts: ["127.0.0.1:5044"]

```

Cheers,  
Pedro Lopes

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 9, 2016, 4:42pm UTC](https://discuss.elastic.co/t/doubts-in-filebeat/55079/2 "2016-07-09T16:42:13Z")

</div>

> But if I add data to the end of a existing/read file, filebeats reads the hole file from start including new line. What I'm looking for is that filebeat reads the new added data for existing files.

_Exactly_ how are you adding the data to the end of a file? Are you using a text editor?

> Is this possible?

Yes, of course. It's Filebeat's main use case.

---

<div class="post-metadata">

**Author:** ![Pedro\_Lopes](https://avatars.discourse-cdn.com/v4/letter/p/e0b2c6/32.png) [@Pedro\_Lopes](https://discuss.elastic.co/u/Pedro_Lopes)\
**Post date:** [July 9, 2016, 5:44pm UTC](https://discuss.elastic.co/t/doubts-in-filebeat/55079/3 "2016-07-09T17:44:55Z")

</div>

Hi Magnus,

The csv's will be write with a service I have, but for testing purposes before implementing Filebeat I'm using sed via command line, weirdly Filebeat processes the file from the beginning!

That's my doubt, is behaving in a not expected way. Any input?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 9, 2016, 8:33pm UTC](https://discuss.elastic.co/t/doubts-in-filebeat/55079/4 "2016-07-09T20:33:01Z")

</div>

_Exactly_ how are you adding the data to the end of a file? "Using sed" is not an answer to that question. What _exact_ command did you use?

---

<div class="post-metadata">

**Author:** ![Pedro\_Lopes](https://avatars.discourse-cdn.com/v4/letter/p/e0b2c6/32.png) [@Pedro\_Lopes](https://discuss.elastic.co/u/Pedro_Lopes)\
**Post date:** [July 10, 2016, 10:40am UTC](https://discuss.elastic.co/t/doubts-in-filebeat/55079/5 "2016-07-10T10:40:53Z")

</div>

Here goes:

sed -i '$ a sample\_text' type\_one1.csv

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 10, 2016, 3:35pm UTC](https://discuss.elastic.co/t/doubts-in-filebeat/55079/6 "2016-07-10T15:35:29Z")

</div>

That results in a new inode number for the file, making Logstash believe that the file is new and needs to be read from the top:

```nohighlight
$ echo 'first line' > testfile
$ ls -li testfile
3412319 -rw-r--r-- 1 magnus magnus 11 Jul 10 17:32 testfile
$ sed -i '$ a sample_text' testfile
$ ls -li testfile
3412320 -rw-r--r-- 1 magnus magnus 23 Jul 10 17:33 testfile

```

Use `echo 'a sample text' >> type_one1.csv` instead.

---

<div class="post-metadata">

**Author:** ![Pedro\_Lopes](https://avatars.discourse-cdn.com/v4/letter/p/e0b2c6/32.png) [@Pedro\_Lopes](https://discuss.elastic.co/u/Pedro_Lopes)\
**Post date:** [July 11, 2016, 4:39pm UTC](https://discuss.elastic.co/t/doubts-in-filebeat/55079/7 "2016-07-11T16:39:05Z")

</div>

Hi Magnus,

Thanks for your tip, it actually worked 🙂 I was kinda noob, anyway solved!

Best regards

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 9:51pm UTC](https://discuss.elastic.co/t/doubts-in-filebeat/55079/8 "2017-07-05T21:51:00Z")

</div>