# Using custom \_id (from Kinesis)

**URL:** <https://discuss.elastic.co/t/using-custom--id-from-kinesis/51936>\
**Category:** Elasticsearch\
**Created:** [June 6, 2016, 12:01pm UTC](https://discuss.elastic.co/t/using-custom--id-from-kinesis/51936 "2016-06-06T12:01:48Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![shushu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shushu/32/10037_2.png) [@shushu](https://discuss.elastic.co/u/shushu)\
**Post date:** [June 6, 2016, 12:01pm UTC](https://discuss.elastic.co/t/using-custom--id-from-kinesis/51936/1 "2016-06-06T12:01:48Z")

</div>

Hello,  
Our app uses Spark job to read from Kinesis, and write to Elastic.  
We got a situation in which the job failed to write to Elastic, but kept reading from Kinesis, which made us missing data.  
Trying to figure out a solution, we thought about using the kinesis checkpoint, which is unique, as the elastic document \_id.  
This will mean, that on any failure, we could just rollback to know checkpoint, and restart the job, which will just overwrite existing documents (if happens).

What do you think ?  
Is using custom id is a proper solution ?  
What about performance of Elastic in this case ?

Thanks,  
Shushu

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 6, 2016, 11:16pm UTC](https://discuss.elastic.co/t/using-custom--id-from-kinesis/51936/2 "2016-06-06T23:16:52Z")

</div>

This could work, but why not handle the failure in your code a bit better?

---

<div class="post-metadata">

**Author:** ![shushu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shushu/32/10037_2.png) [@shushu](https://discuss.elastic.co/u/shushu)\
**Post date:** [June 7, 2016, 6:52am UTC](https://discuss.elastic.co/t/using-custom--id-from-kinesis/51936/3 "2016-06-07T06:52:00Z")

</div>

Making our code is certainly true, and we keep making it better all the time.  
The problem is - stability is still shaky, and if we must keep 100% of the data streaming properly, we can't say "no, our job will never break".  
It is software - it will break for one reason or the other. The question is - how do we revive from the failure.  
So, when you say "this could work" - what does it mean ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:45pm UTC](https://discuss.elastic.co/t/using-custom--id-from-kinesis/51936/4 "2017-07-05T22:45:44Z")

</div>