# Logstash elasticsearch input reads same set of data everytime

**URL:** <https://discuss.elastic.co/t/logstash-elasticsearch-input-reads-same-set-of-data-everytime/236451>\
**Category:** Logstash\
**Created:** [June 10, 2020, 7:29am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-reads-same-set-of-data-everytime/236451 "2020-06-10T07:29:57Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ullas7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ullas7/32/68349_2.png) [@Ullas7](https://discuss.elastic.co/u/Ullas7)\
**Post date:** [June 10, 2020, 7:29am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-reads-same-set-of-data-everytime/236451/1 "2020-06-10T07:29:57Z")

</div>

I am reading data from elasticsearch with below conf

```auto

input {
	 
	 elasticsearch {
        hosts => "localhost:9200"
		index => "indexName"
         query => '{ "sort": ["_doc"] }'
		schedule => "/20 * * * * *"
		docinfo => true
		 
      }
}

filter {
}

output {
 stdout { codec => rubydebug }
   }

```

But every time it prints the same data. I have only two records and I am expected just ones . every 20 seconds its prints same two records  
I tried with setting below conf in logstash.yml

```auto
pipeline.java_execution: false
pipeline.workers: 1

```

Nothing worked for me.

---

<div class="post-metadata">

**Author:** ![Jenni](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jenni/32/29684_2.png) [@Jenni](https://discuss.elastic.co/u/Jenni)\
**Post date:** [June 10, 2020, 8:09am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-reads-same-set-of-data-everytime/236451/2 "2020-06-10T08:09:22Z")

</div>

You are executing the same query every 20 minutes, so you get the same data every 20 minutes – both entries. That's the expected behavior. If you don't want to read these entries again, you'll have to adjust the query accordingly, e.g. with a time range, if there is a field in your entries that indicates their age.

---

<div class="post-metadata">

**Author:** ![Ullas7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ullas7/32/68349_2.png) [@Ullas7](https://discuss.elastic.co/u/Ullas7)\
**Post date:** [June 10, 2020, 9:09am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-reads-same-set-of-data-everytime/236451/3 "2020-06-10T09:09:20Z")

</div>

@Jenni Thanks for quick response. I dont have a timestamp field . This is test data to verify whether logstash keeps tracks of read records,but looks its not.

BTW I am executing in every 20 seconds.  
I am looking something similar to since\_db in elastic input

---

<div class="post-metadata">

**Author:** ![Ullas7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ullas7/32/68349_2.png) [@Ullas7](https://discuss.elastic.co/u/Ullas7)\
**Post date:** [June 10, 2020, 4:12pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-reads-same-set-of-data-everytime/236451/4 "2020-06-10T16:12:22Z")

</div>

Well, I just tried adding sort by **@timestamp** , something like this

```auto
input {
	 
	 elasticsearch {
        hosts => "localhost:9200"
		index => "indexName"
         query => '{ "sort": ["@timestamp"] }'
		schedule => "/20 * * * * *"
		docinfo => true
		 
      }
}

filter {
}

output {
 stdout { codec => rubydebug }
   }

```

Still, the behaviour is the same. Logstash doesn't keep track of read events in case of elasticsearch input plugin? @Badger any help on this?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 8, 2020, 4:12pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-reads-same-set-of-data-everytime/236451/5 "2020-07-08T16:12:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.