# Duplicate data in same index **URL:** \ **Category:** Elasticsearch\ **Created:** [December 9, 2011, 1:40am UTC](https://discuss.elastic.co/t/duplicate-data-in-same-index/6104 "2011-12-09T01:40:01Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![aschaar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aschaar/32/2991_2.png) [@aschaar](https://discuss.elastic.co/u/aschaar)\ **Post date:** [December 9, 2011, 1:40am UTC](https://discuss.elastic.co/t/duplicate-data-in-same-index/6104/1 "2011-12-09T01:40:01Z")
Here's duplicate records: [http://pastebin.mozilla.org/1397732](http://pastebin.mozilla.org/1397732) ES Configuration: [http://oremj.pastebin.mozilla.org/1397404](http://oremj.pastebin.mozilla.org/1397404) Back story: We pushed a new release on Wednesday around 2pm. At that time we deleted the index, setup mapping, and reindexed everything. I watched the results on our search page drop to 0, then climb back up to around 1k as the indexing task completed. Everything looked fine. However, this morning(Thursday) search suddenly was returning twice as many results. Over 2k. We do have a nightly refresh index task that kicks off around 2:30am. Notice the "last\_update": "2011-12-08T02:30:28" value on the second record, but the first record has a "last\_update":"2011-12-07T03:12:17" which is Tuesday night, before we did the release and before we deleted the index. So the question is, how did these records come back from the dead? Replication error? Nodes not all in sync? Let me know if you need any more information. Arron #flightdeck @ [irc.mozilla.org](http://irc.mozilla.org) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 9, 2011, 3:26pm UTC](https://discuss.elastic.co/t/duplicate-data-in-same-index/6104/2 "2011-12-09T15:26:53Z")
Which version are you using? Are you using routing or parent/child mapping? Can you run the same search with explain set to true and gist the result? On Fri, Dec 9, 2011 at 3:40 AM, aschaar [aschaar@gmail.com](mailto:aschaar@gmail.com) wrote: > Here's duplicate records: > > [http://pastebin.mozilla.org/1397732](http://pastebin.mozilla.org/1397732) > > ES Configuration: > [http://oremj.pastebin.mozilla.org/1397404](http://oremj.pastebin.mozilla.org/1397404) > > Back story: > We pushed a new release on Wednesday around 2pm. At that time we deleted > the index, setup mapping, and reindexed everything. I watched the results > on our search page drop to 0, then climb back up to around 1k as the > indexing task completed. Everything looked fine. > > However, this morning(Thursday) search suddenly was returning twice as many > results. Over 2k. We do have a nightly refresh index task that kicks off > around 2:30am. Notice the "last\_update": "2011-12-08T02:30:28" value on > the second record, but the first record has a > "last\_update":"2011-12-07T03:12:17" which is Tuesday night, before we did > the release and before we deleted the index. > > So the question is, how did these records come back from the dead? > Replication error? Nodes not all in sync? > > Let me know if you need any more information. > > Arron > #flightdeck @ [irc.mozilla.org](http://irc.mozilla.org) > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Duplicate-data-in-same-index-tp3571874p3571874.html](http://elasticsearch-users.115913.n3.nabble.com/Duplicate-data-in-same-index-tp3571874p3571874.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![aschaar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aschaar/32/2991_2.png) [@aschaar](https://discuss.elastic.co/u/aschaar)\ **Post date:** [December 9, 2011, 5:38pm UTC](https://discuss.elastic.co/t/duplicate-data-in-same-index/6104/3 "2011-12-09T17:38:28Z")
Version 0.17.4, No routing or parent/child mapping. Gist with explain=true [http://oremj.pastebin.mozilla.org/1398661](http://oremj.pastebin.mozilla.org/1398661) ---
**Author:** ![aschaar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aschaar/32/2991_2.png) [@aschaar](https://discuss.elastic.co/u/aschaar)\ **Post date:** [December 12, 2011, 5:31pm UTC](https://discuss.elastic.co/t/duplicate-data-in-same-index/6104/4 "2011-12-12T17:31:44Z")
Bump... ---
**Author:** ![aschaar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aschaar/32/2991_2.png) [@aschaar](https://discuss.elastic.co/u/aschaar)\ **Post date:** [December 13, 2011, 7:03pm UTC](https://discuss.elastic.co/t/duplicate-data-in-same-index/6104/5 "2011-12-13T19:03:07Z")
Servers: [http://pastebin.mozilla.org/1404356](http://pastebin.mozilla.org/1404356) Here are the results from running a query with explain=true from last week. [http://pastebin.mozilla.org/1404340](http://pastebin.mozilla.org/1404340) Notable differences \_shard: 1, \_node: "wI\_Er7xrQLWd7J1Kdh7Zsw" has the old package of "last\_update": "2011-12-07T03:12:17" \_shard: 4 of each node had the recent package "last\_update": "2011-12-09T02:30:30" Today we ran the query again and here are the results: [http://pastebin.mozilla.org/1404342](http://pastebin.mozilla.org/1404342) Notable differences \_shard: 1 of each node has the old package of "last\_update": "2011-12-12T02:30:48" \_shard: 4, \_node" : "G5EUD-YFTA2-ymrLpTCodA" has the most recent version of the package "last\_update": "2011-12-13T02:37:33" Thoughts? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 13, 2011, 10:28pm UTC](https://discuss.elastic.co/t/duplicate-data-in-same-index/6104/6 "2011-12-13T22:28:50Z")
When did oyu start to use elasticsearch (since which version)? Maybe it was before a version that used the type when hashing (pre 0.13.0)? If it is, then any future version that used the same data should have set cluster.routing.operation.use\_type to true in the settings, otherwise you might get into this situation. On Tue, Dec 13, 2011 at 9:03 PM, aschaar [aschaar@gmail.com](mailto:aschaar@gmail.com) wrote: > Servers: > [http://pastebin.mozilla.org/1404356](http://pastebin.mozilla.org/1404356) > > Here are the results from running a query with explain=true from last week. > [http://pastebin.mozilla.org/1404340](http://pastebin.mozilla.org/1404340) > > Notable differences > > \_shard: 1, \_node: "wI\_Er7xrQLWd7J1Kdh7Zsw" has the old package of > "last\_update": "2011-12-07T03:12:17" > > \_shard: 4 of each node had the recent package "last\_update": > "2011-12-09T02:30:30" > > Today we ran the query again and here are the results: > [http://pastebin.mozilla.org/1404342](http://pastebin.mozilla.org/1404342) > > Notable differences > > \_shard: 1 of each node has the old package of "last\_update": > "2011-12-12T02:30:48" > > \_shard: 4, \_node" : "G5EUD-YFTA2-ymrLpTCodA" has the most recent version of > the package "last\_update": "2011-12-13T02:37:33" > > Thoughts? > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Duplicate-data-in-same-index-tp3571874p3583276.html](http://elasticsearch-users.115913.n3.nabble.com/Duplicate-data-in-same-index-tp3571874p3583276.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:45am UTC](https://discuss.elastic.co/t/duplicate-data-in-same-index/6104/7 "2017-07-06T03:45:29Z")