# Logstash loading same file after restart

**URL:** <https://discuss.elastic.co/t/logstash-loading-same-file-after-restart/101123>\
**Category:** Logstash\
**Created:** [September 20, 2017, 7:42am UTC](https://discuss.elastic.co/t/logstash-loading-same-file-after-restart/101123 "2017-09-20T07:42:23Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![DILIP\_SHARMA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dilip_sharma/32/42672_2.png) [@DILIP\_SHARMA](https://discuss.elastic.co/u/DILIP_SHARMA)\
**Post date:** [September 20, 2017, 7:42am UTC](https://discuss.elastic.co/t/logstash-loading-same-file-after-restart/101123/1 "2017-09-20T07:42:23Z")

</div>

Hi all ,

whenever my logstash start , its load the same file again , this causing duplicate data in EC .  
this is my config file -

```
    input {
	file {
	path => "E:\Local_Elasticsearch\logstashv5\datafiles/*.csv"
	start_position => "beginning"
	sincedb_path => "/dev/null"
	ignore_older => 0
	}
}

filter {
  csv {
      separator => ","
     columns => ["A","B","C","D","E","F"]
  }
  mutate {convert =>[F" , "integer"]}

  date {
      match => ["B", "ISO8601", "YYYY-MM-dd HH:mm:ss"]
      target => "B"
      locale => "en"
    }
	
 date {
      match => ["D", "ISO8601", "YYYY-MM-dd HH:mm:ss"]
      target => "D"
      locale => "en"
    }
	
 date {
      match => ["E", "ISO8601", "YYYY-MM-dd HH:mm:ss"]
      target => "E"
      locale => "en"
    }
  }

output {
	elasticsearch {
		hosts => "localhost"
		index => "testindex1"
		document_type => "tesdata1"
	}
	stdout{}

}

```

and my csv file

A,B,C,D,E,F  
A1,2017/09/16 00:00:00,3U0604,2017/09/16 19:22:00,2017/09/16 19:30:00,1  
A2,2017/09/16 00:00:00,3U0604,2017/09/16 19:22:00,2017/09/16 19:30:00,2  
A3,2017/09/16 00:00:00,9W0527,2017/09/16 11:59:00,2017/09/16 12:10:00,3  
A4,2017/09/16 00:00:00,9W0531,2017/09/16 02:45:00,2017/09/16 02:35:00,4  
A5,2017/09/16 00:00:00,9W0535,2017/09/16 14:56:00,2017/09/16 15:10:00,5  
A6,2017/09/16 00:00:00,9W0537,2017/09/16 18:30:00,2017/09/16 18:35:00,6  
A7,2017/09/16 00:00:00,9W0541,2017/09/16 12:05:00,2017/09/16 12:00:00,7

any suggestion to solve this issue .  
Thanks

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 20, 2017, 8:08am UTC](https://discuss.elastic.co/t/logstash-loading-same-file-after-restart/101123/2 "2017-09-20T08:08:50Z")

</div>

> [@DILIP\_SHARMA](#):
>
> sincedb\_path =\> "/dev/null"

This causes Logstash to not keep track of which files that have been processed, meaning that all files will be processed after every restart.

---

<div class="post-metadata">

**Author:** ![DILIP\_SHARMA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dilip_sharma/32/42672_2.png) [@DILIP\_SHARMA](https://discuss.elastic.co/u/DILIP_SHARMA)\
**Post date:** [September 20, 2017, 8:56am UTC](https://discuss.elastic.co/t/logstash-loading-same-file-after-restart/101123/3 "2017-09-20T08:56:01Z")

</div>

should i give file name instead of null where its store track or i should remove this line ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 20, 2017, 9:02am UTC](https://discuss.elastic.co/t/logstash-loading-same-file-after-restart/101123/4 "2017-09-20T09:02:22Z")

</div>

Removing the line will use the default location for the synced file. You can specify a file of you want to control the location.

---

<div class="post-metadata">

**Author:** ![DILIP\_SHARMA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dilip_sharma/32/42672_2.png) [@DILIP\_SHARMA](https://discuss.elastic.co/u/DILIP_SHARMA)\
**Post date:** [September 20, 2017, 10:23am UTC](https://discuss.elastic.co/t/logstash-loading-same-file-after-restart/101123/5 "2017-09-20T10:23:35Z")

</div>

Thanks ..

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 18, 2017, 10:23am UTC](https://discuss.elastic.co/t/logstash-loading-same-file-after-restart/101123/6 "2017-10-18T10:23:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.