# Logstash elasticsearch input plugin logs duplication

**URL:** <https://discuss.elastic.co/t/logstash-elasticsearch-input-plugin-logs-duplication/144700>\
**Category:** Logstash\
**Created:** [August 16, 2018, 11:01am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-plugin-logs-duplication/144700 "2018-08-16T11:01:28Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![mevimi](https://avatars.discourse-cdn.com/v4/letter/m/c6cbf5/32.png) [@mevimi](https://discuss.elastic.co/u/mevimi)\
**Post date:** [August 16, 2018, 11:01am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-plugin-logs-duplication/144700/1 "2018-08-16T11:01:28Z")

</div>

Hi,  
I use logstash 5.6.8 with the below logstash configuration to forward logs from my elasticsearch to syslog server. I have scheduled the inut plugin to read every minute. I can see same old logs are being read every minute and sent to the syslog server. How can i avoid this duplication of logs?

```
   input {
   elasticsearch 
   id => "_logs"
    hosts => "localhost:9200"
    index => "_audit_logs-*"
    query => '{ "query": { "query_string": { "query": "*" } } }'
    size => 500
    scroll => "5m"
    docinfo => false
	schedule => "* * * * *"
  }
}

output {
	syslog{
		host => "localhost"
		port => 601
	}
}

```

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 16, 2018, 11:06am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-plugin-logs-duplication/144700/2 "2018-08-16T11:06:05Z")

</div>

The elasticsearch input doesn't have any functionality for skipping already processed documents so there's no simple way of avoiding duplicates with the design you've chosen.

How do the documents end up in ES? Would it be possible to hook into the pipeline earlier on?

---

<div class="post-metadata">

**Author:** ![mevimi](https://avatars.discourse-cdn.com/v4/letter/m/c6cbf5/32.png) [@mevimi](https://discuss.elastic.co/u/mevimi)\
**Post date:** [August 16, 2018, 11:09am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-plugin-logs-duplication/144700/3 "2018-08-16T11:09:26Z")

</div>

Logs are provided by a team in Elasticsearch from multiple sources. I do not have control over this. But i am allowed to read from the Elasticsearch. So i am trying to implement logstash to read from Elasticsearch and forward it to syslog server.

Is there nothing like the sincedb for file input which keeps a track of the last record?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 16, 2018, 11:12am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-plugin-logs-duplication/144700/4 "2018-08-16T11:12:14Z")

</div>

> Logs are provided by a team in Elasticsearch from multiple sources. I do not have control over this. But i am allowed to read from the Elasticsearch. So i am trying to implement logstash to read from Elasticsearch and forward it to syslog server.

That's a flawed architecture. Don't use ES as a message-passing mechanism.

> Is there nothing like the sincedb for file input which keeps a track of the last record?

No.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 13, 2018, 11:12am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-plugin-logs-duplication/144700/5 "2018-09-13T11:12:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.