# Logstash does not read rewritable csv file well

**URL:** <https://discuss.elastic.co/t/logstash-does-not-read-rewritable-csv-file-well/267880>\
**Category:** Logstash\
**Created:** [March 20, 2021, 10:03am UTC](https://discuss.elastic.co/t/logstash-does-not-read-rewritable-csv-file-well/267880 "2021-03-20T10:03:42Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Andrew\_Foxis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrew_foxis/32/69811_2.png) [@Andrew\_Foxis](https://discuss.elastic.co/u/Andrew_Foxis)\
**Post date:** [March 20, 2021, 10:03am UTC](https://discuss.elastic.co/t/logstash-does-not-read-rewritable-csv-file-well/267880/1 "2021-03-20T10:03:42Z")

</div>

Hi all.

I have csv file:

```
addr;peer;port;name;datetime (headers)
172.12.10.1; 34.15.67.43; 1123; peter; 2021-03-15 00:02:34 ( value rows (100-200 rows))

```

This file rewritable every 5 minutes.

I want to get the full contents of the file every 5 minutes.  
But instead, I sometimes get the full contents of the file, sometimes the last line, sometimes part of the last line. There are no \_csvfailure tags

Why is this happening? Please help me with the correct configuration

I confgure basic pipeline for logstash:

```auto
    input {
        file {
    		type => "clients"
            path => "/opt/clients/clients-out.csv"
            sincedb_path => "/dev/null"
    		start_position => "beginning"
        }

    }

    filter {
    	if [type] == "clients"	{
    		
    	csv {
    			separator => ";"
                columns => ['addr','peer','port','name','datetime']
    			skip_header => true
    	 }

    	mutate {
    		add_tag => ["clients"]
    	}
    	}
    }

    output {
    	if "clients" in [tags] {
    			elasticsearch {
    				hosts => "localhost:9200"
    				index => "clients-%{+YYYY.MM.dd}"
    				manage_template => true
    				ilm_enabled => false
    				ssl => true
    				ssl_certificate_verification => "false"	
    			}
    	}
    }

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 20, 2021, 4:20pm UTC](https://discuss.elastic.co/t/logstash-does-not-read-rewritable-csv-file-well/267880/2 "2021-03-20T16:20:59Z")

</div>

Setting sincedb\_path to "/dev/null" prevents logstash from persisting the in-memory sincedb to disk across restarts. However, the in-memory db is still maintained. So if a file is re-written with the same inode only the part that is longer than the original file will be read. If the file is re-written with a different inode then the whole file will be read.

You may be able to use read mode, and delete the file every time it is read, but you may still have problems with [inode re-use](https://github.com/logstash-plugins/logstash-input-file/issues/251). That is actually fairly easily [fixable](https://github.com/logstash-plugins/logstash-input-file/issues/213), but it has not been done.

Figuring out whether you have already read a file is a _really_ hard problem, far harder than you might think at first, and the file input uses a cheap algorithm that is usually right.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 17, 2021, 4:21pm UTC](https://discuss.elastic.co/t/logstash-does-not-read-rewritable-csv-file-well/267880/3 "2021-04-17T16:21:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.