Logstash elasticsearch input reads same set of data everytime

Elastic Stack Logstash
1

I am reading data from elasticsearch with below conf


input {
	 
	 elasticsearch {
        hosts => "localhost:9200"
		index => "indexName"
         query => '{ "sort": [ "_doc" ] }'
		schedule => "/20 * * * * *"
		docinfo => true
		 
      }
}

filter {
}

output {
 stdout { codec => rubydebug }
   }

But every time it prints the same data. I have only two records and I am expected just ones . every 20 seconds its prints same two records
I tried with setting below conf in logstash.yml

pipeline.java_execution: false
pipeline.workers: 1

Nothing worked for me.

2

You are executing the same query every 20 minutes, so you get the same data every 20 minutes – both entries. That's the expected behavior. If you don't want to read these entries again, you'll have to adjust the query accordingly, e.g. with a time range, if there is a field in your entries that indicates their age.

1 Like
3

@Jenni Thanks for quick response. I dont have a timestamp field . This is test data to verify whether logstash keeps tracks of read records,but looks its not.

BTW I am executing in every 20 seconds.
I am looking something similar to since_db in elastic input

4

Well, I just tried adding sort by @timestamp , something like this

input {
	 
	 elasticsearch {
        hosts => "localhost:9200"
		index => "indexName"
         query => '{ "sort": [ "@timestamp" ] }'
		schedule => "/20 * * * * *"
		docinfo => true
		 
      }
}

filter {
}

output {
 stdout { codec => rubydebug }
   }

Still, the behaviour is the same. Logstash doesn't keep track of read events in case of elasticsearch input plugin? @Badger any help on this?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.