Doubts in Filebeat


(Pedro Lopes) #1

Hello guys,

I have a filebeat.yml that works perfectly sending csv files to logstash, and then logstash to elastic.

If I add a new file to any of the folders filebeats starts it's magic. But if I add data to the end of a existing/read file, filebeats reads the hole file from start including new line. What I'm looking for is that filebeat reads the new added data for existing files. Is this possible?

Here is my filebeat.yml:

filebeat:
  prospectors:
    -
      paths:
        - "/home/vagrant/files/type_one*.csv"
      document_type: type_one
      ignore_older: 1m
    -
      paths:
        - "/home/vagrant/files/type_two*.csv"
      document_type: type_two
      ignore_older: 1m
    -
      paths:
        - "/home/vagrant/files/type_three*.csv"
      document_type: type_three
      ignore_older: 1m
output:
  logstash:
    hosts: ["127.0.0.1:5044"]

Cheers,
Pedro Lopes


(Magnus Bäck) #2

But if I add data to the end of a existing/read file, filebeats reads the hole file from start including new line. What I'm looking for is that filebeat reads the new added data for existing files.

Exactly how are you adding the data to the end of a file? Are you using a text editor?

Is this possible?

Yes, of course. It's Filebeat's main use case.


(Pedro Lopes) #3

Hi Magnus,

The csv's will be write with a service I have, but for testing purposes before implementing Filebeat I'm using sed via command line, weirdly Filebeat processes the file from the beginning!

That's my doubt, is behaving in a not expected way. Any input?


(Magnus Bäck) #4

Exactly how are you adding the data to the end of a file? "Using sed" is not an answer to that question. What exact command did you use?


(Pedro Lopes) #5

Here goes:

sed -i '$ a sample_text' type_one1.csv


(Magnus Bäck) #6

That results in a new inode number for the file, making Logstash believe that the file is new and needs to be read from the top:

$ echo 'first line' > testfile
$ ls -li testfile
3412319 -rw-r--r-- 1 magnus magnus 11 Jul 10 17:32 testfile
$ sed -i '$ a sample_text' testfile
$ ls -li testfile
3412320 -rw-r--r-- 1 magnus magnus 23 Jul 10 17:33 testfile

Use echo 'a sample text' >> type_one1.csv instead.


(Pedro Lopes) #7

Hi Magnus,

Thanks for your tip, it actually worked :slight_smile: I was kinda noob, anyway solved!

Best regards


(system) #8