# Documents with duplicate \_id in an index split by time?

**URL:** https://discuss.elastic.co/t/documents-with-duplicate-id-in-an-index-split-by-time/248260
**Category:** Elasticsearch
**Created:** [September 11, 2020, 3:58am UTC](https://discuss.elastic.co/t/documents-with-duplicate-id-in-an-index-split-by-time/248260 "2020-09-11T03:58:20Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![mykael](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@mykael](https://discuss.elastic.co/u/mykael)
#### Post date: [September 11, 2020, 3:58am UTC](https://discuss.elastic.co/t/documents-with-duplicate-id-in-an-index-split-by-time/248260/1 "2020-09-11T03:58:20Z")

</div>

Might just be me misunderstanding how ids and indexing work here.

I've got a stream of records and I write them to an index - my\_data

Now, to keep it manigable, because there are lots of records, I'm telling logstash to split the index into segments by time:

```
elasticsearch {
  hosts => ["localhost"]
  document_id => "%{id}"
  index => "my_data-%{+YYYY.MM}" 
}

```

I then define an index\_pattern of my\_data-\* to drive Kibana.

I've found a case where I initially got the timestamp on a record wrong and when I re-indexed it with the correct timestamp, instead of it replacing the old version with the same ID (the timestamp isn't in the id), it simply created a second copy in the index segment that corresponded to the new timestamp.

Is there anything I have to do to get the id to be unique across all of the index segments? I really need it to be unique within the index pattern...

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 11, 2020, 5:05am UTC](https://discuss.elastic.co/t/documents-with-duplicate-id-in-an-index-split-by-time/248260/2 "2020-09-11T05:05:10Z")

</div>

Document ids are unique within a single index only. You can not make them unique across multiple indices linked by an index pattern.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 9, 2020, 5:05am UTC](https://discuss.elastic.co/t/documents-with-duplicate-id-in-an-index-split-by-time/248260/3 "2020-10-09T05:05:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.