# Reload the same file from the beginning without restarting logstash

**URL:** <https://discuss.elastic.co/t/reload-the-same-file-from-the-beginning-without-restarting-logstash/68156>\
**Category:** Logstash\
**Created:** [December 6, 2016, 10:54am UTC](https://discuss.elastic.co/t/reload-the-same-file-from-the-beginning-without-restarting-logstash/68156 "2016-12-06T10:54:18Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![mehdi-aouadi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehdi-aouadi/32/13644_2.png) [@mehdi-aouadi](https://discuss.elastic.co/u/mehdi-aouadi)\
**Post date:** [December 6, 2016, 10:54am UTC](https://discuss.elastic.co/t/reload-the-same-file-from-the-beginning-without-restarting-logstash/68156/1 "2016-12-06T10:54:18Z")

</div>

I managed to force Logstash reloading the whole file by pointing the `sincedb_path` to `NUL` (Windows environment) and setting the `start_position` at the beginning. Here is my `file input` configuration:

```
input {
     file {
        path => "myfile.csv"
        start_position => beginning
        ignore_older => 0
        type => "my_document_type"
        sincedb_path => "NUL"
        stat_interval => 1
    }
}

```

The file is actually reloaded every time I restart Logstash and every time it is modified, but I want it to be reloaded each second as mentioned in `stat_interval`.  
I also need it to be reloaded even if there is no modification and without restarting logstash because I am adding a date based field in the filters and I need the same data every day with an updated `date_field` :

```
filter {
    csv {
        columns => ["MyFirstColumn", "MySecondColumn"]
        separator => ";"
        add_field => {
        "date_field" => "%{+ddMMyyy}"
        }
    }
}  

```

Here is an example of the expected behavior :

File content :

```
Column A;Column B
Value X;Value Y  

```

Data sent to Elastic search index :

```
Column A : Value X, Column B : Value Y, date_field : 05122016

```

The day after, even without modifying the file I want the following data to be added to the same index in Elasticsearch :

```
Column A : Value X, Column B : Value Y, date_field : 06122016
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 6, 2016, 11:40am UTC](https://discuss.elastic.co/t/reload-the-same-file-from-the-beginning-without-restarting-logstash/68156/2 "2016-12-06T11:40:32Z")

</div>

I don't think the file input is the best fit here. Why not use the exec plugin instead?

---

<div class="post-metadata">

**Author:** ![mehdi-aouadi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehdi-aouadi/32/13644_2.png) [@mehdi-aouadi](https://discuss.elastic.co/u/mehdi-aouadi)\
**Post date:** [December 7, 2016, 8:10am UTC](https://discuss.elastic.co/t/reload-the-same-file-from-the-beginning-without-restarting-logstash/68156/3 "2016-12-07T08:10:31Z")

</div>

Thanks for the tip. I tried using a cat command on the file, the problem now is that it loads the content of the whole file as a single block. Here is an example of the file content :

```
myFirstColumn;mySecondColumn;mythirdColumn
valueA;ValueB;ValueC
Value1;Value2;Value3
ValueX;ValueY;ValueZ

```

And here is my config :

```
input {
	exec {
		command => "cat myfile.csv"
		interval => 2
		add_field => {
			  "tag" => "mytag"
		}
	}
}
filter {

	if [tag] == "mytag" {
		csv {
			columns => ["myFirstColumn", "mySecondColumn", "mythirdColumn"]
			separator => ";"		
		}
}  

```

It sends the whole content of the file without splitting it content. Is something missing ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 7, 2016, 8:16am UTC](https://discuss.elastic.co/t/reload-the-same-file-from-the-beginning-without-restarting-logstash/68156/4 "2016-12-07T08:16:07Z")

</div>

No, that's expected. You can a split filter to split events on newlines.

---

<div class="post-metadata">

**Author:** ![mehdi-aouadi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehdi-aouadi/32/13644_2.png) [@mehdi-aouadi](https://discuss.elastic.co/u/mehdi-aouadi)\
**Post date:** [December 7, 2016, 9:36am UTC](https://discuss.elastic.co/t/reload-the-same-file-from-the-beginning-without-restarting-logstash/68156/6 "2016-12-07T09:36:29Z")

</div>

I added a split filter before the csv one and it now working well.  
To sumarize, here is my config file :

```
input {
	exec {
		command => "cat myfile.csv"
		interval => 2
		add_field => {
			  "tag" => "mytag"
		}
	}
}
filter {

	if [tag] == "mytag" {
                split {
                        terminator => "\n"
                }
		csv {
			columns => ["myFirstColumn", "mySecondColumn", "mythirdColumn"]
			separator => ";"		
		}
}
output {
	if [tag] == "mytag" {
		elasticsearch {
			hosts => ["localhost:9200"]
			index => "myIndex"
                }
	}
} 

```

Thank you for your help 😊

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 4, 2017, 9:36am UTC](https://discuss.elastic.co/t/reload-the-same-file-from-the-beginning-without-restarting-logstash/68156/7 "2017-01-04T09:36:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.