Logstash does not read rewritable csv file well

[Andrew_Foxis]

Hi all.

I have csv file:

addr;peer;port;name;datetime (headers)
172.12.10.1; 34.15.67.43; 1123; peter; 2021-03-15 00:02:34 ( value rows (100-200 rows))

This file rewritable every 5 minutes.

I want to get the full contents of the file every 5 minutes.
But instead, I sometimes get the full contents of the file, sometimes the last line, sometimes part of the last line. There are no _csvfailure tags

Why is this happening? Please help me with the correct configuration

I confgure basic pipeline for logstash:

    input {
        file {
    		type => "clients"
            path => "/opt/clients/clients-out.csv"
            sincedb_path => "/dev/null"
    		start_position => "beginning"
        }

    }

    filter {
    	if [type] == "clients"	{
    		
    	csv {
    			separator => ";"
                columns => ['addr','peer','port','name','datetime']
    			skip_header => true
    	 }

    	mutate {
    		add_tag => [ "clients"]
    	}
    	}
    }

    output {
    	if "clients" in [tags] {
    			elasticsearch {
    				hosts => "localhost:9200"
    				index => "clients-%{+YYYY.MM.dd}"
    				manage_template => true
    				ilm_enabled => false
    				ssl => true
    				ssl_certificate_verification => "false"	
    			}
    	}
    }

[Badger]

Setting sincedb_path to "/dev/null" prevents logstash from persisting the in-memory sincedb to disk across restarts. However, the in-memory db is still maintained. So if a file is re-written with the same inode only the part that is longer than the original file will be read. If the file is re-written with a different inode then the whole file will be read.

You may be able to use read mode, and delete the file every time it is read, but you may still have problems with inode re-use. That is actually fairly easily fixable, but it has not been done.

Figuring out whether you have already read a file is a really hard problem, far harder than you might think at first, and the file input uses a cheap algorithm that is usually right.