# Retrieve ES index with LS input plugin NOT(!) everytime from the beginning?

**URL:** <https://discuss.elastic.co/t/retrieve-es-index-with-ls-input-plugin-not-everytime-from-the-beginning/69009>\
**Category:** Logstash\
**Created:** [December 14, 2016, 11:45am UTC](https://discuss.elastic.co/t/retrieve-es-index-with-ls-input-plugin-not-everytime-from-the-beginning/69009 "2016-12-14T11:45:41Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![fxiger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fxiger/32/12670_2.png) [@fxiger](https://discuss.elastic.co/u/fxiger)\
**Post date:** [December 14, 2016, 11:45am UTC](https://discuss.elastic.co/t/retrieve-es-index-with-ls-input-plugin-not-everytime-from-the-beginning/69009/1 "2016-12-14T11:45:41Z")

</div>

Hi,  
i'm currently retrieving data from an "external" elastic index and write the events into csv files (for later postprocessing). This works fine so far.

**My problem is:**  
When i shutdown LS and restart it at a later point, it continiues reading from the beginning of the existing index, so i will get duplicate events.

**I'm using (on Ubuntu 16.04):**  
logstash 2.4.1  
logstash-filter-csv (2.1.3)  
logstash-input-file (2.2.5)  
logstash-output-elasticsearch (2.7.1)

**My config looks like the following:**

> input {  
> elasticsearch {  
> hosts =\> ["somehost:9200"]  
> index =\> "someindex"  
> type =\> "om\_event"  
> }  
> }

> filter {  
> if [type] == "om\_event" {  
> [...]  
> }  
> }

> output {  
> if [type] == "om\_event" {  
> csv {  
> csv\_options =\> {"col\_sep" =\> ";"}  
> fields =\> ['some','fields']  
> path =\> "/path/to/file/sometest\_%{+yyyy.MM.dd}.log"  
> }  
> stdout {  
> codec =\> rubydebug  
> }  
> }  
> }

I googled a bit and someone mentioned to use a timestamp in a query within the input-section like below:

> query =\> '{"query":{"range":{"parsed\_date":{"gte": "${LAST\_RUN}"}}}}'

so i tried to create a bash-script which measures the timestamp of my last run and writes it to a logfile on each run and also sets it as an environment variable. But it seems, i've started diving in this parent/child-world of linux processes without fully understanding it ☹

> #!/bin/bash

> # read last run from file
> 
> source ./logstash\_last\_run.log

> echo "previous last run: $LAST\_RUN"

> # call logstash
> 
> /opt/logstash/bin/logstash -f /etc/logstash/conf.d/om\_elastic.conf

> # store new date in logfile
> 
> export LAST\_RUN=$(date +"%FT%T")  
> echo "export LAST\_RUN=$LAST\_RUN" \> logstash\_last\_run.log  
> echo "new last run: $LAST\_RUN"

Does anybody have an idea how to solve this problem?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 11, 2017, 11:45am UTC](https://discuss.elastic.co/t/retrieve-es-index-with-ls-input-plugin-not-everytime-from-the-beginning/69009/2 "2017-01-11T11:45:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.