# Duplicate records in elastic search during load of records with Logstash

**URL:** <https://discuss.elastic.co/t/duplicate-records-in-elastic-search-during-load-of-records-with-logstash/217372>\
**Category:** Logstash\
**Created:** [January 31, 2020, 12:41pm UTC](https://discuss.elastic.co/t/duplicate-records-in-elastic-search-during-load-of-records-with-logstash/217372 "2020-01-31T12:41:09Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ramypala](https://avatars.discourse-cdn.com/v4/letter/r/43a26b/32.png) [@ramypala](https://discuss.elastic.co/u/ramypala)\
**Post date:** [January 31, 2020, 12:41pm UTC](https://discuss.elastic.co/t/duplicate-records-in-elastic-search-during-load-of-records-with-logstash/217372/1 "2020-01-31T12:41:09Z")

</div>

Hi Team,

I am loading a ServiceNow table data into Elastic search using Logstash config with HTTP Poller Plugin.For example if the table has 40 records,after the load of 40 records in Elastic search, data load is not getting stopped.Same set of 40 records gets loaded repeatedly generating duplicating records in Elastic search index.I tried to give document\_id="%{number}" in output plugin under elastic search in config file.number is unique change number in my table.But only one record is loaded in index when I checked in Kibana.Please help on this.

# Config File:

input {  
http\_poller {  
urls =\> {  
url =\> "[https://dev63285.service-now.com/change\_request\_list.do?JSONv2&display\_value=True&sysparm\_exclude\_reference\_link=True&sysparm\_fields=number%2Cstate%2Copened\_at%2Cclosed\_at%2Cpriority%2Cassigned\_to%2Cassignment\_group%2Cactive%2Cimpact%2Curgency%2Cseverity&sysparm\_limit=1sysparm\_view=json\_view](https://dev63285.service-now.com/change_request_list.do?JSONv2&display_value=True&sysparm_exclude_reference_link=True&sysparm_fields=number%2Cstate%2Copened_at%2Cclosed_at%2Cpriority%2Cassigned_to%2Cassignment_group%2Cactive%2Cimpact%2Curgency%2Cseverity&sysparm_limit=1sysparm_view=json_view)"  
}  
connect\_timeout =\> 1000000  
request\_timeout =\> 900000  
socket\_timeout =\> 180  
user =\> "XXX"  
password =\> "XXX"  
schedule =\> { cron =\> "\* \* \* \* \*"}  
codec =\> "json"  
metadata\_target =\> "http\_poller\_metadata"  
}  
}  
filter  
{  
split {  
field =\> "records"  
}  
}  
output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> "servicenowinc"  
}  
stdout {  
codec =\> rubydebug  
}  
}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 28, 2020, 12:41pm UTC](https://discuss.elastic.co/t/duplicate-records-in-elastic-search-during-load-of-records-with-logstash/217372/2 "2020-02-28T12:41:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.