# Logstash neither detect the newly added data in CSV file nor push it into elasticsearch

**URL:** <https://discuss.elastic.co/t/logstash-neither-detect-the-newly-added-data-in-csv-file-nor-push-it-into-elasticsearch/174938>\
**Category:** Logstash\
**Created:** [April 2, 2019, 8:12am UTC](https://discuss.elastic.co/t/logstash-neither-detect-the-newly-added-data-in-csv-file-nor-push-it-into-elasticsearch/174938 "2019-04-02T08:12:13Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![cheriemilk](https://avatars.discourse-cdn.com/v4/letter/c/c37758/32.png) [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)\
**Post date:** [April 2, 2019, 8:12am UTC](https://discuss.elastic.co/t/logstash-neither-detect-the-newly-added-data-in-csv-file-nor-push-it-into-elasticsearch/174938/1 "2019-04-02T08:12:13Z")

</div>

Hi team,

I have a CSV file which has 3 records at the beginning and parsed by logstash successfully. (I didn't close logstash at this moment)

Later, I open the file and add forth records and want logstash detect the new record automatically. But seems logstash failed to detected it and in kibana, there's still 3 records displayed.

Here is the CSV file. (the last record is newly added)

```
       GIT_ORG,GIT_REPOS,COMMIT_SHA1,COMMIT_AUTHOR,COMMIT_DATE
bizx,au-recruiting,6739c82bcf830b05d4d36e9fd715bc5715e0c380,Kaderjan Ilghar,2018-01-24
bizx,idl-analytics-api,1be44f52f25f6b540f284eb17e8cee5838826cb9,ssheriff,2018-01-23
bizx,idl-analytics-api,1be44f52f25f6b540f284eb17e8cee5838826cb9,ssheriff,2018-01-22
bizx,idl-analytics-api,1be44f52f25f6b540f284eb17e8cee5838826cb9,ssheriff,2018-01-21

```

Here is the logstash.conf

```
input { file{ path => "C:/elkstack/elasticsearch-6.5.1/logs/test1.csv"		
              start_position => "beginning"
			  sincedb_path => "C:/elkstack/elasticsearch-6.5.1/sincedb/sincedb.txt" }
	  }

 filter { 

    date { match => ["COMMIT_DATE", "yyyy-MM-dd"]
           target => "@timestamp" }
               
    if "_dateparsefailure" in [tags] { drop {} }

    csv { columns => [ "GIT_ORG",
                        "GIT_REPOS",
                        "COMMIT_SHA1",
                        "COMMIT_AUTHOR",
						"COMMIT_DATE"]
           separator => ","
		   skip_header => "true"
		   }   
   }

output {
    elasticsearch {
	 action => "index"
	 hosts => "localhost:9200"
	 index => "test1"
     manage_template => true
     template => "C:/elkstack/elasticsearch-6.5.1/indextemp/dtemplate.json"
     template_name=> "dtemplate"
     template_overwrite => true }

stdout { codec => rubydebug {metadata => true}}

```

}

What's the problem here?

---

<div class="post-metadata">

**Author:** ![cheriemilk](https://avatars.discourse-cdn.com/v4/letter/c/c37758/32.png) [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)\
**Post date:** [April 2, 2019, 8:40am UTC](https://discuss.elastic.co/t/logstash-neither-detect-the-newly-added-data-in-csv-file-nor-push-it-into-elasticsearch/174938/2 "2019-04-02T08:40:01Z")

</div>

1 observation.

1. If i open the CSV with excel to add the new record and save. logstash doesn't detect the change.
2. If I open the CSV file with sublime to add the new record and save. logstash can detect the change.

For 1. I use notepadd++ to open it and configure to display the end of line. Below is the screenshot! Does it because I didn't start a new line???

 ![1](https://us1.discourse-cdn.com/elastic/original/3X/3/1/31298f555c4ab65e6a164fc1d50d70f50bd8e554.jpeg)

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 2, 2019, 1:21pm UTC](https://discuss.elastic.co/t/logstash-neither-detect-the-newly-added-data-in-csv-file-nor-push-it-into-elasticsearch/174938/3 "2019-04-02T13:21:30Z")

</div>

I cannot speak to why logstash does not notice when Excel updates the file, but the COMMIT\_DATE field is created by the csv filter. It will not exist when the date filter executes, so that filter is a no-op. Move the csv filter to be before the date filter.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 30, 2019, 1:28pm UTC](https://discuss.elastic.co/t/logstash-neither-detect-the-newly-added-data-in-csv-file-nor-push-it-into-elasticsearch/174938/4 "2019-04-30T13:28:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.