# Setting a long custom \_id value when inserting into Elastic search

**URL:** <https://discuss.elastic.co/t/setting-a-long-custom-id-value-when-inserting-into-elastic-search/266300>\
**Category:** Elasticsearch\
**Created:** [March 4, 2021, 11:42pm UTC](https://discuss.elastic.co/t/setting-a-long-custom-id-value-when-inserting-into-elastic-search/266300 "2021-03-04T23:42:18Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![scott\_stash](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@scott\_stash](https://discuss.elastic.co/u/scott_stash)\
**Post date:** [March 4, 2021, 11:42pm UTC](https://discuss.elastic.co/t/setting-a-long-custom-id-value-when-inserting-into-elastic-search/266300/1 "2021-03-04T23:42:18Z")

</div>

I'm reading data with filebeat, sending to logstash and it ends up in ES. One issue we have is sometimes Filebeat re-reads files leading to duplicate entries in Elastic Search.

One idea was to create a custom \_id value in logstash based on the filename + offset. I would assume that the filename rarely is greater then 32 characters, the files we process are generally 500MB or less, so we are looking at up to another 9 digits. So in most cases the id\_value would be 30-40 characters. (Perhaps we could hash this to make it smaller)

Anyone know of any performance issues, or done anything similar?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 1, 2021, 11:42pm UTC](https://discuss.elastic.co/t/setting-a-long-custom-id-value-when-inserting-into-elastic-search/266300/2 "2021-04-01T23:42:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.