# Duplicate data in ES **URL:** \ **Category:** Elasticsearch\ **Created:** [December 15, 2010, 10:10pm UTC](https://discuss.elastic.co/t/duplicate-data-in-es/3669 "2010-12-15T22:10:36Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Anurag\_Phadke](https://avatars.discourse-cdn.com/v4/letter/a/5f9b8f/32.png) [@Anurag\_Phadke](https://discuss.elastic.co/u/Anurag_Phadke)\ **Post date:** [December 15, 2010, 10:10pm UTC](https://discuss.elastic.co/t/duplicate-data-in-es/3669/1 "2010-12-15T22:10:36Z")
Hello all, ES maintains one copy of each json for a given index/type. I ran the following search query: http://\/logs/builds/\_search?pretty=true&q=mozilla-central-linux-debug-1292432368-mochitest-other.gz and it returned the following jsons: { "\_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 2, "max\_score" : 1.5457389, "hits" : [ { "\_index" : "logs", "\_type" : "builds", "\_id" : "qiZEn2fMQBSKHAbA2UDa7A", "\_score" : 1.5457389, "\_source" : { "builds" : {"machine": "talos-r3-fed-049", "buildtype": "debug", "testrun\_count": 4, "buildid": "20101215085928", "builder": "mozilla-central\_fedora-debug\_test-mochitest-other", "filename": "mozilla-central-linux-debug-1292432368-mochitest-other.gz", "repo": "mozilla-central", "platform": "linux", "buildurl": "[http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1292432368/firefox-4.0b9pre.en-US.linux-i686.tar.bz2](http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1292432368/firefox-4.0b9pre.en-US.linux-i686.tar.bz2)", "starttime": "1292436935", "revision": "d0234003e042"} } }, { "\_index" : "logs", "\_type" : "builds", "\_id" : "gNBDktUJTieNNPvO2Jn\_qg", "\_score" : 1.4803665, "\_source" : { "builds" : {"machine": "talos-r3-fed-049", "buildtype": "debug", "testrun\_count": 4, "buildid": "20101215085928", "builder": "mozilla-central\_fedora-debug\_test-mochitest-other", "filename": "mozilla-central-linux-debug-1292432368-mochitest-other.gz", "repo": "mozilla-central", "platform": "linux", "buildurl": "[http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1292432368/firefox-4.0b9pre.en-US.linux-i686.tar.bz2](http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1292432368/firefox-4.0b9pre.en-US.linux-i686.tar.bz2)", "starttime": "1292436935", "revision": "d0234003e042"} } } ] } } both the jsons look same (to me atleast).... any idea why this might be happening? -anurag ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 15, 2010, 10:43pm UTC](https://discuss.elastic.co/t/duplicate-data-in-es/3669/2 "2010-12-15T22:43:15Z")
ES will only update docs if they have the same id. If you are indexing into elasticsearch without specifying an id, then one will be automatically be created and it is considered as new doc. In your case, there were probably two docs indexed with this content. If you have something that can uniquely identify the document, then you can provide it as the id, and then documents will be updated if they already exists. On Thu, Dec 16, 2010 at 12:10 AM, Anurag [anurag.phadke@gmail.com](mailto:anurag.phadke@gmail.com) wrote: > Hello all, > ES maintains one copy of each json for a given index/type. I ran the > following search query: > http:// > \/logs/builds/\_search?pretty=true&q=mozilla-central-linux-debug-1292432368-mochitest-other.gz > > and it returned the following jsons: > { > "\_shards" : { > "total" : 5, > "successful" : 5, > "failed" : 0 > }, > "hits" : { > "total" : 2, > "max\_score" : 1.5457389, > "hits" : [ { > "\_index" : "logs", > "\_type" : "builds", > "\_id" : "qiZEn2fMQBSKHAbA2UDa7A", > "\_score" : 1.5457389, "\_source" : { "builds" : {"machine": > "talos-r3-fed-049", "buildtype": "debug", "testrun\_count": 4, > "buildid": "20101215085928", "builder": > "mozilla-central\_fedora-debug\_test-mochitest-other", "filename": > "mozilla-central-linux-debug-1292432368-mochitest-other.gz", "repo": > "mozilla-central", "platform": "linux", "buildurl": > " > [http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1292432368/firefox-4.0b9pre.en-US.linux-i686.tar.bz2](http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1292432368/firefox-4.0b9pre.en-US.linux-i686.tar.bz2) > ", > "starttime": "1292436935", "revision": "d0234003e042"} } > }, { > "\_index" : "logs", > "\_type" : "builds", > "\_id" : "gNBDktUJTieNNPvO2Jn\_qg", > "\_score" : 1.4803665, "\_source" : { "builds" : {"machine": > "talos-r3-fed-049", "buildtype": "debug", "testrun\_count": 4, > "buildid": "20101215085928", "builder": > "mozilla-central\_fedora-debug\_test-mochitest-other", "filename": > "mozilla-central-linux-debug-1292432368-mochitest-other.gz", "repo": > "mozilla-central", "platform": "linux", "buildurl": > " > [http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1292432368/firefox-4.0b9pre.en-US.linux-i686.tar.bz2](http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1292432368/firefox-4.0b9pre.en-US.linux-i686.tar.bz2) > ", > "starttime": "1292436935", "revision": "d0234003e042"} } > } ] > } > } > > both the jsons look same (to me atleast).... any idea why this might > be happening? > > -anurag ---
**Author:** ![Anurag\_Phadke](https://avatars.discourse-cdn.com/v4/letter/a/5f9b8f/32.png) [@Anurag\_Phadke](https://discuss.elastic.co/u/Anurag_Phadke)\ **Post date:** [December 15, 2010, 10:56pm UTC](https://discuss.elastic.co/t/duplicate-data-in-es/3669/3 "2010-12-15T22:56:56Z")
Shay, The data is getting indexed via multiple puts without an id (from our end). We don't have a way of maintaining an id internally, would it be possible to have a getAndPut call that compares the contents of whole json and only inserts a new one inside the index if it's unique? We can probably do this by having two API calls for an individual json, not sure how this can be done when for multiple inserts via single or two API calls? -anurag On Dec 15, 2010, at 2:43 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > ES will only update docs if they have the same id. If you are indexing into elasticsearch without specifying an id, then one will be automatically be created and it is considered as new doc. In your case, there were probably two docs indexed with this content. > > If you have something that can uniquely identify the document, then you can provide it as the id, and then documents will be updated if they already exists. > > On Thu, Dec 16, 2010 at 12:10 AM, Anurag [anurag.phadke@gmail.com](mailto:anurag.phadke@gmail.com) wrote: > Hello all, > ES maintains one copy of each json for a given index/type. I ran the > following search query: > http://\/logs/builds/\_search?pretty=true&q=mozilla-central-linux-debug-1292432368-mochitest-other.gz > > and it returned the following jsons: > { > "\_shards" : { > "total" : 5, > "successful" : 5, > "failed" : 0 > }, > "hits" : { > "total" : 2, > "max\_score" : 1.5457389, > "hits" : [ { > "\_index" : "logs", > "\_type" : "builds", > "\_id" : "qiZEn2fMQBSKHAbA2UDa7A", > "\_score" : 1.5457389, "\_source" : { "builds" : {"machine": > "talos-r3-fed-049", "buildtype": "debug", "testrun\_count": 4, > "buildid": "20101215085928", "builder": > "mozilla-central\_fedora-debug\_test-mochitest-other", "filename": > "mozilla-central-linux-debug-1292432368-mochitest-other.gz", "repo": > "mozilla-central", "platform": "linux", "buildurl": > "[http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1292432368/firefox-4.0b9pre.en-US.linux-i686.tar.bz2](http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1292432368/firefox-4.0b9pre.en-US.linux-i686.tar.bz2)", > "starttime": "1292436935", "revision": "d0234003e042"} } > }, { > "\_index" : "logs", > "\_type" : "builds", > "\_id" : "gNBDktUJTieNNPvO2Jn\_qg", > "\_score" : 1.4803665, "\_source" : { "builds" : {"machine": > "talos-r3-fed-049", "buildtype": "debug", "testrun\_count": 4, > "buildid": "20101215085928", "builder": > "mozilla-central\_fedora-debug\_test-mochitest-other", "filename": > "mozilla-central-linux-debug-1292432368-mochitest-other.gz", "repo": > "mozilla-central", "platform": "linux", "buildurl": > "[http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1292432368/firefox-4.0b9pre.en-US.linux-i686.tar.bz2](http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1292432368/firefox-4.0b9pre.en-US.linux-i686.tar.bz2)", > "starttime": "1292436935", "revision": "d0234003e042"} } > } ] > } > } > > both the jsons look same (to me atleast).... any idea why this might > be happening? > > -anurag ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 15, 2010, 11:02pm UTC](https://discuss.elastic.co/t/duplicate-data-in-es/3669/4 "2010-12-15T23:02:50Z")
There is no way to get a document based on source content matching. You can md5 the document, and check against it. A strategy to do get then index is problematic since it will really slow down indexing, and the near real time aspect will be problematic. On Thu, Dec 16, 2010 at 12:57 AM, Anurag Phadke [anurag.phadke@gmail.com](mailto:anurag.phadke@gmail.com)wrote: > Shay, > The data is getting indexed via multiple puts without an id (from our end). > We don't have a way of maintaining an id internally, would it be possible to > have a getAndPut call that compares the contents of whole json and only > inserts a new one inside the index if it's unique? > We can probably do this by having two API calls for an individual json, not > sure how this can be done when for multiple inserts via single or two API > calls? > > -anurag > > On Dec 15, 2010, at 2:43 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) > wrote: > > ES will only update docs if they have the same id. If you are indexing into > elasticsearch without specifying an id, then one will be automatically be > created and it is considered as new doc. In your case, there were probably > two docs indexed with this content. > > If you have something that can uniquely identify the document, then you can > provide it as the id, and then documents will be updated if they already > exists. > > On Thu, Dec 16, 2010 at 12:10 AM, Anurag \< [anurag.phadke@gmail.com](mailto:anurag.phadke@gmail.com) > [anurag.phadke@gmail.com](mailto:anurag.phadke@gmail.com)\> wrote: > > > Hello all, > > ES maintains one copy of each json for a given index/type. I ran the > > following search query: > > http:// > > \/logs/builds/\_search?pretty=true&q=mozilla-central-linux-debug-1292432368-mochitest-other.gz > > > > and it returned the following jsons: > > { > > "\_shards" : { > > "total" : 5, > > "successful" : 5, > > "failed" : 0 > > }, > > "hits" : { > > "total" : 2, > > "max\_score" : 1.5457389, > > "hits" : [ { > > "\_index" : "logs", > > "\_type" : "builds", > > "\_id" : "qiZEn2fMQBSKHAbA2UDa7A", > > "\_score" : 1.5457389, "\_source" : { "builds" : {"machine": > > "talos-r3-fed-049", "buildtype": "debug", "testrun\_count": 4, > > "buildid": "20101215085928", "builder": > > "mozilla-central\_fedora-debug\_test-mochitest-other", "filename": > > "mozilla-central-linux-debug-1292432368-mochitest-other.gz", "repo": > > "mozilla-central", "platform": "linux", "buildurl": > > "[http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1292432368/firefox-4.0b9pre.en-US.linux-i686.tar.bz2](http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1292432368/firefox-4.0b9pre.en-US.linux-i686.tar.bz2) > > [http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1292432368/firefox-4.0b9pre.en-US.linux-i686.tar.bz2](http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1292432368/firefox-4.0b9pre.en-US.linux-i686.tar.bz2) > > ", > > "starttime": "1292436935", "revision": "d0234003e042"} } > > }, { > > "\_index" : "logs", > > "\_type" : "builds", > > "\_id" : "gNBDktUJTieNNPvO2Jn\_qg", > > "\_score" : 1.4803665, "\_source" : { "builds" : {"machine": > > "talos-r3-fed-049", "buildtype": "debug", "testrun\_count": 4, > > "buildid": "20101215085928", "builder": > > "mozilla-central\_fedora-debug\_test-mochitest-other", "filename": > > "mozilla-central-linux-debug-1292432368-mochitest-other.gz", "repo": > > "mozilla-central", "platform": "linux", "buildurl": > > "[http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1292432368/firefox-4.0b9pre.en-US.linux-i686.tar.bz2](http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1292432368/firefox-4.0b9pre.en-US.linux-i686.tar.bz2) > > [http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1292432368/firefox-4.0b9pre.en-US.linux-i686.tar.bz2](http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux-debug/1292432368/firefox-4.0b9pre.en-US.linux-i686.tar.bz2) > > ", > > "starttime": "1292436935", "revision": "d0234003e042"} } > > } ] > > } > > } > > > > both the jsons look same (to me atleast).... any idea why this might > > be happening? > > > > -anurag ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:15am UTC](https://discuss.elastic.co/t/duplicate-data-in-es/3669/5 "2017-07-06T04:15:15Z")