Doubts in Filebeat

Elastic Stack Beats
1

Hello guys,

I have a filebeat.yml that works perfectly sending csv files to logstash, and then logstash to elastic.

If I add a new file to any of the folders filebeats starts it's magic. But if I add data to the end of a existing/read file, filebeats reads the hole file from start including new line. What I'm looking for is that filebeat reads the new added data for existing files. Is this possible?

Here is my filebeat.yml:

filebeat:
  prospectors:
    -
      paths:
        - "/home/vagrant/files/type_one*.csv"
      document_type: type_one
      ignore_older: 1m
    -
      paths:
        - "/home/vagrant/files/type_two*.csv"
      document_type: type_two
      ignore_older: 1m
    -
      paths:
        - "/home/vagrant/files/type_three*.csv"
      document_type: type_three
      ignore_older: 1m
output:
  logstash:
    hosts: ["127.0.0.1:5044"]

Cheers,
Pedro Lopes

2

But if I add data to the end of a existing/read file, filebeats reads the hole file from start including new line. What I'm looking for is that filebeat reads the new added data for existing files.

Exactly how are you adding the data to the end of a file? Are you using a text editor?

Is this possible?

Yes, of course. It's Filebeat's main use case.

3

Hi Magnus,

The csv's will be write with a service I have, but for testing purposes before implementing Filebeat I'm using sed via command line, weirdly Filebeat processes the file from the beginning!

That's my doubt, is behaving in a not expected way. Any input?

4

Exactly how are you adding the data to the end of a file? "Using sed" is not an answer to that question. What exact command did you use?

5

Here goes:

sed -i '$ a sample_text' type_one1.csv

6

That results in a new inode number for the file, making Logstash believe that the file is new and needs to be read from the top:

$ echo 'first line' > testfile
$ ls -li testfile
3412319 -rw-r--r-- 1 magnus magnus 11 Jul 10 17:32 testfile
$ sed -i '$ a sample_text' testfile
$ ls -li testfile
3412320 -rw-r--r-- 1 magnus magnus 23 Jul 10 17:33 testfile

Use echo 'a sample text' >> type_one1.csv instead.

7

Hi Magnus,

Thanks for your tip, it actually worked :slight_smile: I was kinda noob, anyway solved!

Best regards