Reading file from beginning using file input

Elastic Stack Logstash
1

I have this setting, but it is not reading file again and again.
I want to read this file from beginning when it is changed.
I am getting this file copy from another system every one hour.

It reads once when I starts logstash from command line. Then as test when I transfer file or update that file it is not re-reading at all.

input {
file {
path => "/elkdata01/a_size_final_log"
sincedb_path => "/dev/null"
start_position => "beginning"
}
}

If I touch/edit this log file manually on /elkdata01/ then it reads it. but if I scp that file from different system it does not. Why? what do I have to do to make it work

2

Even if you set sincedb_path => "/dev/null" the file input still uses an in-memory sincedb. It just does not persist it to disk. It will only re-read a file with that name from the beginning if the inode number changes. So instead of overwriting it, you would need to move it aside and write a new file.

3

ok. I deleted file. and then scp new file (same_name) but it didn't read it.

Then I did vi of that file on logstash server and it read it right away.

4

even if touch the file on logstash server it does not read it.
only way right now it re-reads file is if I do vi, change value in it and save it.

5

If you delete a file and create a new one, then on some filesystems the inode number is re-used. That will prevent the file input from seeing it as a new file. For example

echo foo > foo ; ls -li foo ; rm foo 
echo foo > foo ; ls -li foo ; rm foo

gets me

4272123 -rw-r--r--. 1 user user 4 Feb 13 11:41 foo
4272123 -rw-r--r--. 1 user user 4 Feb 13 11:41 foo

It re-uses inode 4272123. Ugh!

6

Alright make sense now. this is what I am gone a do.
create a file with day of week somewhere.

delete all log file once a week. and rotate number

foo.1 foo.2 foo3 ...foo.7
and rotate.

I tested this method with some dummy file name and it worked.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.