# 1.2.0 routing issue and explicit routing **URL:** \ **Category:** Elasticsearch\ **Created:** [June 10, 2014, 9:44pm UTC](https://discuss.elastic.co/t/1-2-0-routing-issue-and-explicit-routing/18028 "2014-06-10T21:44:07Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Ralph\_Meijer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ralph_meijer/32/1483_2.png) [@Ralph\_Meijer](https://discuss.elastic.co/u/Ralph_Meijer)\ **Post date:** [June 10, 2014, 9:44pm UTC](https://discuss.elastic.co/t/1-2-0-routing-issue-and-explicit-routing/18028/1 "2014-06-10T21:44:07Z")
Hi, I use explicit routing in some of my logging indexes, and I have a _lot_ of data. The tool provided to fix the routing issue that showed up in 1.2.0 takes forever on my indexes, but while it has been running, I'm wondering if I even have the problem sketched by the (very informative) blog item. I don't seem to have lost access to events, and am now getting reports of duplicate events. So, to make this really clear: - Does using explicit routing (in this cased defined in mapping to use a customer id field) have the same routing issue? - Does the tool take explicit routing into account, or am I now messing things up royally? - If so, can I remove those duplicates, while keeping my explicit routing? - All this might be useful to mention in an update to the blog post. I've shutdown the tool for the time being. -- Cheers, ralphm -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/27ccbc63-8fb6-4020-9b41-1ad8855426d8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/27ccbc63-8fb6-4020-9b41-1ad8855426d8%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![brian\_yoder](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)\ **Post date:** [June 10, 2014, 10:14pm UTC](https://discuss.elastic.co/t/1-2-0-routing-issue-and-explicit-routing/18028/2 "2014-06-10T22:14:38Z")
As an aside, I am also wondering why this link [http://www.elasticsearch.org/downloads/1-2-0/](http://www.elasticsearch.org/downloads/1-2-0/) is still active and available when it was supposed to be pulled. Brian -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1a0a4bd7-2149-482a-b502-b13a830ca7b2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1a0a4bd7-2149-482a-b502-b13a830ca7b2%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 10, 2014, 10:18pm UTC](https://discuss.elastic.co/t/1-2-0-routing-issue-and-explicit-routing/18028/3 "2014-06-10T22:18:03Z")
Probably because it contains the release notes etc. You can't download any of the files from the links, though a note about it being removed would be handy I guess. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 11 June 2014 08:14, Brian [brian.from.fl@gmail.com](mailto:brian.from.fl@gmail.com) wrote: > As an aside, I am also wondering why this link > [http://www.elasticsearch.org/downloads/1-2-0/](http://www.elasticsearch.org/downloads/1-2-0/) is still active and > available when it was supposed to be pulled. > > Brian > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/1a0a4bd7-2149-482a-b502-b13a830ca7b2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1a0a4bd7-2149-482a-b502-b13a830ca7b2%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/1a0a4bd7-2149-482a-b502-b13a830ca7b2%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1a0a4bd7-2149-482a-b502-b13a830ca7b2%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624Yg6DDWjTvpMr94hOocm0SPjs1RdSVFrsH%2Bo7UKWE7\_gA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624Yg6DDWjTvpMr94hOocm0SPjs1RdSVFrsH%2Bo7UKWE7_gA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [June 11, 2014, 8:50am UTC](https://discuss.elastic.co/t/1-2-0-routing-issue-and-explicit-routing/18028/4 "2014-06-11T08:50:49Z")
On Tue, Jun 10, 2014 at 11:44 PM, Ralph Meijer [ralphm@ik.nu](mailto:ralphm@ik.nu) wrote: > - Does using explicit routing (in this cased defined in mapping to use a > customer id field) have the same routing issue? Yes. The issue is on the hashing function, so it occurs when using explicit routing as well. > - Does the tool take explicit routing into account, or am I now messing > things up royally? Yes it does. When you use custom routing, either explicitely by setting the routing value in the mappings or indexing requests, or implicitely in case of parent/child, Elasticsearch stores a special `_routing` value that the tool uses in order to identify the right shard for this document. - If so, can I remove those duplicates, while keeping my explicit routing? > Yes: the tool has a `delete` action to remove documents that have been routed to the wrong shard. -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6faxqZkLH7ghO7%3DkkWXgWp7ZXPQKG9xVTtTV-n1zcmoA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6faxqZkLH7ghO7%3DkkWXgWp7ZXPQKG9xVTtTV-n1zcmoA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ralph\_Meijer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ralph_meijer/32/1483_2.png) [@Ralph\_Meijer](https://discuss.elastic.co/u/Ralph_Meijer)\ **Post date:** [June 12, 2014, 6:53am UTC](https://discuss.elastic.co/t/1-2-0-routing-issue-and-explicit-routing/18028/5 "2014-06-12T06:53:03Z")
On 2014-06-11 10:50, Adrien Grand wrote: > On Tue, Jun 10, 2014 at 11:44 PM, Ralph Meijer [ralphm@ik.nu](mailto:ralphm@ik.nu) wrote: > > > - Does using explicit routing (in this cased defined in mapping to use a > > customer id field) have the same routing issue? > > Yes. The issue is on the hashing function, so it occurs when using explicit > routing as well. > [..] Ok, thanks! I've been running the tool, but looking at my graphs, it looks like its copy\_if\_missing function processes about 8 documents a second, which seems really slow compared to the index rates I normally have. Because this is just one daily index with over a 100 million entries, completing the sequence is going to take forever. Is there a way to speed this up? If not, I'll probably stop the process and reindex using the \_source field, taking the copies for granted. -- ralphm -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/53994E4F.1090704%40ik.nu](https://groups.google.com/d/msgid/elasticsearch/53994E4F.1090704%40ik.nu). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [June 12, 2014, 7:23am UTC](https://discuss.elastic.co/t/1-2-0-routing-issue-and-explicit-routing/18028/6 "2014-06-12T07:23:45Z")
How many of these 100M documents have been indexed with Elasticsearch 1.2.0? The tool would only send indexing requests for documents that have been misrouted, so if you only see 8 indexing requests per second, which is indeed low, that might mean that the bottleneck is searching for the mis-routed documents. On Thu, Jun 12, 2014 at 8:53 AM, Ralph Meijer [ralphm@ik.nu](mailto:ralphm@ik.nu) wrote: > On 2014-06-11 10:50, Adrien Grand wrote: > > > On Tue, Jun 10, 2014 at 11:44 PM, Ralph Meijer [ralphm@ik.nu](mailto:ralphm@ik.nu) wrote: > > > > > - Does using explicit routing (in this cased defined in mapping to use > > > a > > > customer id field) have the same routing issue? > > > > Yes. The issue is on the hashing function, so it occurs when using > > explicit > > routing as well. > > [..] > > Ok, thanks! > > I've been running the tool, but looking at my graphs, it looks like its > copy\_if\_missing function processes about 8 documents a second, which > seems really slow compared to the index rates I normally have. Because > this is just one daily index with over a 100 million entries, completing > the sequence is going to take forever. Is there a way to speed this up? > > If not, I'll probably stop the process and reindex using the \_source > field, taking the copies for granted. > > -- > ralphm > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/53994E4F.1090704%40ik.nu](https://groups.google.com/d/msgid/elasticsearch/53994E4F.1090704%40ik.nu). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j75UgRQPt4OqTFX0jFY85kNThVbxo5x\_Ka9bFNc\_zNaaQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j75UgRQPt4OqTFX0jFY85kNThVbxo5x_Ka9bFNc_zNaaQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ralph\_Meijer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ralph_meijer/32/1483_2.png) [@Ralph\_Meijer](https://discuss.elastic.co/u/Ralph_Meijer)\ **Post date:** [June 12, 2014, 7:31am UTC](https://discuss.elastic.co/t/1-2-0-routing-issue-and-explicit-routing/18028/7 "2014-06-12T07:31:53Z")
On 2014-06-12 09:23, Adrien Grand wrote: > How many of these 100M documents have been indexed with Elasticsearch > 1.2.0? All of them, and I have like 40 of them, with their size a similar order of magnitude :-/ > The tool would only send indexing requests for documents that have > been misrouted, so if you only see 8 indexing requests per second, which is > indeed low, that might mean that the bottleneck is searching for the > mis-routed documents. I couldn't find the source of the tool on GitHub, but maybe if I had some insight into how that search works, I could change some settings. Ideas welcome, of course. If it helps, I'm in #elasticsearch as ralphm. -- ralphm -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/53995769.5010500%40ik.nu](https://groups.google.com/d/msgid/elasticsearch/53995769.5010500%40ik.nu). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [June 12, 2014, 8:33am UTC](https://discuss.elastic.co/t/1-2-0-routing-issue-and-explicit-routing/18028/8 "2014-06-12T08:33:50Z")
On Thu, Jun 12, 2014 at 9:31 AM, Ralph Meijer [ralphm@ik.nu](mailto:ralphm@ik.nu) wrote: > On 2014-06-12 09:23, Adrien Grand wrote: > > > How many of these 100M documents have been indexed with Elasticsearch > > 1.2.0? > > All of them, and I have like 40 of them, with their size a similar order > of magnitude :-/ If all of them were indexed in 1.2.0 then it makes sense to just reindex from \_source as you suggested. > > The tool would only send indexing requests for documents that have > > been misrouted, so if you only see 8 indexing requests per second, which > > is > > indeed low, that might mean that the bottleneck is searching for the > > mis-routed documents. > > I couldn't find the source of the tool on GitHub, but maybe if I had > some insight into how that search works, I could change some settings. > Ideas welcome, of course. If it helps, I'm in #elasticsearch as ralphm. It does a SCAN search that filters using a script filter (the one you added to your config/scripts). And for each matching documents, it reindexes it to the right shard, either by requiring a create operation (copy\_if\_missing) or overriding a potential document that would have the same \_id (copy\_overwrite). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j585BOmxdcjmiQV9vsPPfhiOd7z2wxL3Je5gcTRX5NVfw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j585BOmxdcjmiQV9vsPPfhiOd7z2wxL3Je5gcTRX5NVfw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ralph\_Meijer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ralph_meijer/32/1483_2.png) [@Ralph\_Meijer](https://discuss.elastic.co/u/Ralph_Meijer)\ **Post date:** [June 12, 2014, 8:43am UTC](https://discuss.elastic.co/t/1-2-0-routing-issue-and-explicit-routing/18028/9 "2014-06-12T08:43:35Z")
On 2014-06-12 10:33, Adrien Grand wrote: > On Thu, Jun 12, 2014 at 9:31 AM, Ralph Meijer [ralphm@ik.nu](mailto:ralphm@ik.nu) wrote: > > > On 2014-06-12 09:23, Adrien Grand wrote: > > > > > How many of these 100M documents have been indexed with Elasticsearch > > > 1.2.0? > > > > All of them, and I have like 40 of them, with their size a similar order > > of magnitude :-/ > > If all of them were indexed in 1.2.0 then it makes sense to just reindex > from \_source as you suggested. Ok, thanks! -- ralphm -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/53996837.3080709%40ik.nu](https://groups.google.com/d/msgid/elasticsearch/53996837.3080709%40ik.nu). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:23am UTC](https://discuss.elastic.co/t/1-2-0-routing-issue-and-explicit-routing/18028/10 "2017-07-06T01:23:04Z")