# Elastic\> Logstash\>Elastic dup messages

**URL:** <https://discuss.elastic.co/t/elastic-logstash-elastic-dup-messages/134672>\
**Category:** Logstash\
**Created:** [June 5, 2018, 6:06pm UTC](https://discuss.elastic.co/t/elastic-logstash-elastic-dup-messages/134672 "2018-06-05T18:06:42Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sahana\_athreya](https://avatars.discourse-cdn.com/v4/letter/s/73ab20/32.png) [@Sahana\_athreya](https://discuss.elastic.co/u/Sahana_athreya)\
**Post date:** [June 5, 2018, 6:06pm UTC](https://discuss.elastic.co/t/elastic-logstash-elastic-dup-messages/134672/1 "2018-06-05T18:06:42Z")

</div>

Hi,  
I am trying to fetch data from elastic, add some fields and send it back to elastic on a new index.  
I want to make logstash continuously listen to elastic input plugin.  
I figured that's not possible.  
Now I set a cron to run the logstash every minute.  
But it sends all the data from the beginning to end.  
Is there a way to make it remember the last message sent?

In File input plugin we have start\_position, sincedb\_path and ignore\_older that we can set to do this.  
Are there any options like these in Elasticsearch plugin, in order to make logstash remember the last message and push it to elastic only from the next?

our config looks like this:  
input {  
elasticsearch {  
hosts =\> ["dxvrfpesrch001:9200"]  
user =\> "user"  
password =\> "password"  
index =\> "tradetech-ctp"  
}  
}  
filter {  
json {  
source =\> "message"  
}  
ruby {  
code =\> "event.set('Latency',event.get('sourceTimestamp')-event.get('msgReceivedTimestamp'))"  
}  
ruby {  
code =\> "event.set('unixdate',event.get('timeStamp').to\_i / 1000000)"  
}  
date {  
match =\> ["unixdate", "UNIX\_MS"]  
timezone =\> "UTC"  
}

}  
output {  
elasticsearch {  
hosts =\> ["dxvrfpesrch001:9200"]  
user =\> "user"  
password =\> "password"  
index =\> "tradetech-ctp-seq"  
}  
stdout {  
codec =\> rubydebug  
}  
}

Thank you

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 3, 2018, 6:06pm UTC](https://discuss.elastic.co/t/elastic-logstash-elastic-dup-messages/134672/2 "2018-07-03T18:06:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.