# More than one document with the same \_id **URL:** \ **Category:** Elasticsearch\ **Created:** [April 3, 2019, 2:01pm UTC](https://discuss.elastic.co/t/more-than-one-document-with-the-same-id/175220 "2019-04-03T14:01:57Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![zvonimir](https://avatars.discourse-cdn.com/v4/letter/z/41988e/32.png) [@zvonimir](https://discuss.elastic.co/u/zvonimir)\ **Post date:** [April 3, 2019, 2:01pm UTC](https://discuss.elastic.co/t/more-than-one-document-with-the-same-id/175220/1 "2019-04-03T14:01:57Z")
Hello. I have encountered strange issue on ES 5.6.4 and Java 8. Multiple documents with the same \_id and some with missing \_version are result when using scripted\_upsert to insert and merge multiple events into single document. All documents are indexed into same index and shard on the same node. I am not sure how is this possible and any hint how to troubleshoot is appreciated. Bellow is the example of query and result. ``` GET /_search { "version": true, "_source": ["DatabaseId", "SequenceId"], "query": { "bool": { "must": [ { "match": { "DatabaseId": 1202 } }, { "match": { "SequenceId": 239761506 } } ] } } } { "took": 111, "timed_out": false, "_shards": { "total": 48, "successful": 48, "skipped": 0, "failed": 0 }, "hits": { "total": 4, "max_score": 2, "hits": [ { "_index": "some-index", "_type": "message", "_id": "1202-239761506", "_version": 1, "_score": 2, "_source": { "DatabaseId": 1202, "SequenceId": 239761506 } }, { "_index": "some-index", "_type": "message", "_id": "1202-239761506", "_version": 1, "_score": 2, "_source": { "DatabaseId": 1202, "SequenceId": 239761506 } }, { "_index": "some-index", "_type": "message", "_id": "1202-239761506", "_score": 2, "_source": { "DatabaseId": 1202, "SequenceId": 239761506 } }, { "_index": "some-index", "_type": "message", "_id": "1202-239761506", "_version": 1, "_score": 2, "_source": { "DatabaseId": 1202, "SequenceId": 239761506 } } ] } } ``` Best regards, Zvonimir ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 3, 2019, 2:18pm UTC](https://discuss.elastic.co/t/more-than-one-document-with-the-same-id/175220/2 "2019-04-03T14:18:01Z")
How many shards does the `some-index` index have? Have you used routing while you have been indexing and/or updating? What result do you get if you run the query and use the document id seen here as routing key? ---
**Author:** ![zvonimir](https://avatars.discourse-cdn.com/v4/letter/z/41988e/32.png) [@zvonimir](https://discuss.elastic.co/u/zvonimir)\ **Post date:** [April 3, 2019, 2:30pm UTC](https://discuss.elastic.co/t/more-than-one-document-with-the-same-id/175220/3 "2019-04-03T14:30:28Z")
Index has 48 shards and we are not using custom routing. When searching by \_id, ES returns only single document. Result is the last and correct version of that document. Also when trying to delete older versions I get following error. Looks like older version were not remove correctly from the shard. ``` { "error": { "root_cause": [ { "type": "action_request_validation_exception", "reason": "Validation Failed: 1: illegal version value [-1] for version type [INTERNAL];" } ], "type": "action_request_validation_exception", "reason": "Validation Failed: 1: illegal version value [-1] for version type [INTERNAL];" }, "status": 400 } ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 3, 2019, 2:34pm UTC](https://discuss.elastic.co/t/more-than-one-document-with-the-same-id/175220/4 "2019-04-03T14:34:19Z")
Did you run the search that returned multiple documents with the document id as routing key? ---
**Author:** ![zvonimir](https://avatars.discourse-cdn.com/v4/letter/z/41988e/32.png) [@zvonimir](https://discuss.elastic.co/u/zvonimir)\ **Post date:** [April 4, 2019, 7:18am UTC](https://discuss.elastic.co/t/more-than-one-document-with-the-same-id/175220/5 "2019-04-04T07:18:12Z")
I am not sure what you mean. This query returns only the latest version of document. ``` GET some-index/message/313-10354627132 ``` It appears that root cause of the issue is that older version wasn't properly deleted when document was updated with newer version. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 4, 2019, 7:29am UTC](https://discuss.elastic.co/t/more-than-one-document-with-the-same-id/175220/6 "2019-04-04T07:29:48Z")
That is a GET request that is automatically routed to a single shard. What does the following give? ```auto GET /_search?routing=1202-239761506 { "version": true, "_source": ["DatabaseId", "SequenceId"], "query": { "bool": { "must": [ { "match": { "DatabaseId": 1202 } }, { "match": { "SequenceId": 239761506 } } ] } } } ``` ---
**Author:** ![zvonimir](https://avatars.discourse-cdn.com/v4/letter/z/41988e/32.png) [@zvonimir](https://discuss.elastic.co/u/zvonimir)\ **Post date:** [April 4, 2019, 7:31am UTC](https://discuss.elastic.co/t/more-than-one-document-with-the-same-id/175220/7 "2019-04-04T07:31:49Z")
It returns 4 documents like in my first post. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 4, 2019, 8:14am UTC](https://discuss.elastic.co/t/more-than-one-document-with-the-same-id/175220/8 "2019-04-04T08:14:27Z")
Have never seen that before so will have to leave that for someone else. What does your cluster configuration and hardware look like? ---
**Author:** ![zvonimir](https://avatars.discourse-cdn.com/v4/letter/z/41988e/32.png) [@zvonimir](https://discuss.elastic.co/u/zvonimir)\ **Post date:** [April 4, 2019, 8:29am UTC](https://discuss.elastic.co/t/more-than-one-document-with-the-same-id/175220/9 "2019-04-04T08:29:38Z")
Cluster has 3 dedicated master nodes and 48 data nodes. Indexes have 48 shards and there are ~250 shards per node. Indexing is done in bulk with ~25k/sec rate and it is using painless scripted\_upsert. When trying to reindex those documents to a new index I get following. ``` POST /_reindex { "source":{ "index":"some-index", "query":{ "bool":{ "must":[ { "match":{ "DatabaseId":1202 } }, { "match":{ "SequenceId":239761506 } } ] } } }, "dest":{ "index":"broken-docs" } } { "took": 520, "timed_out": false, "total": 4, "updated": 3, "created": 1, "deleted": 0, "batches": 1, "version_conflicts": 0, "noops": 0, "retries": { "bulk": 0, "search": 0 }, "throttled_millis": 0, "requests_per_second": -1, "throttled_until_millis": 0, "failures": [] } GET broken-docs/_count { "count": 1, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 } } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 2, 2019, 8:29am UTC](https://discuss.elastic.co/t/more-than-one-document-with-the-same-id/175220/10 "2019-05-02T08:29:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.