# Checking for sparse statistics and problems with shard routing **URL:** \ **Category:** Elasticsearch\ **Created:** [October 29, 2015, 8:48am UTC](https://discuss.elastic.co/t/checking-for-sparse-statistics-and-problems-with-shard-routing/33227 "2015-10-29T08:48:05Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Vaughn\_Dickson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vaughn_dickson/32/5588_2.png) [@Vaughn\_Dickson](https://discuss.elastic.co/u/Vaughn_Dickson)\ **Post date:** [October 29, 2015, 8:48am UTC](https://discuss.elastic.co/t/checking-for-sparse-statistics-and-problems-with-shard-routing/33227/1 "2015-10-29T08:48:05Z")
Hi, We have a search index consisting of 206735 web pages (7gb) stored on a Found 2 node cluster with 5 shards. Our \_id is the url of the page, so it seems like each domain will get routed to the same shard? Will this cause us problems with relevancy scoring? I don't fully understand the sparse statistics problem yet. Should we be using custom routing to shuffle documents across shards more, and/or use less shards? Kind regards, Vaughn Dickson ---
**Author:** ![mainec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mainec/32/5557_2.png) [@mainec](https://discuss.elastic.co/u/mainec)\ **Post date:** [October 29, 2015, 10:24am UTC](https://discuss.elastic.co/t/checking-for-sparse-statistics-and-problems-with-shard-routing/33227/2 "2015-10-29T10:24:41Z")
[quote="Vaughn\_Dickson, post:1, topic:33227"] We have a search index consisting of 206735 web pages (7gb) stored on a Found 2 node cluster with 5 shards. Our \_id is the url of the page, so it seems like each domain will get routed to the same shard?[/quote] Reading [\_routing field | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-routing-field.html) I don't think so unless with "URL of the page" you really mean "domain name of the page". > [@Vaughn\_Dickson](#): > > Will this cause us problems with relevancy scoring? Thinking about this question independently of whether or not your particular setup actually suffers from this issue here's what IMHO happens when sharding web pages by domain name: - For scoring you only run into issues if the distribution of your terms is highly uneven. For more information on how this affects scoring see here [Understanding "Query Then Fetch" vs "DFS Query Then Fetch" | Elastic Blog](https://www.elastic.co/blog/understanding-query-then-fetch-vs-dfs-query-then-fetch) - If we are talking about indexing crawled web pages you will run into different problem though: Looking at the number of pages per domain there's plenty of domains out there that publish only a handful of pages. However there's a few domains (think [yahoo.com](http://yahoo.com) and the like) that publish tons of pages. As a result I imagine you'd be running into the "one big user" issue described here: [One Big User | Elasticsearch: The Definitive Guide [2.x] | Elastic](https://www.elastic.co/guide/en/elasticsearch/guide/current/one-big-user.html) Hope this helps, Isabel ---
**Author:** ![Vaughn\_Dickson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vaughn_dickson/32/5588_2.png) [@Vaughn\_Dickson](https://discuss.elastic.co/u/Vaughn_Dickson)\ **Post date:** [October 30, 2015, 7:50am UTC](https://discuss.elastic.co/t/checking-for-sparse-statistics-and-problems-with-shard-routing/33227/3 "2015-10-30T07:50:18Z")
Thanks so much Isabel! That blog post really helped. Our \_id is the full URL to the webpage and not just the domain, so I'm guessing we'll have a fairly decent spread across our shards. I'll investigate the hashing algo just in case. And the global document frequency stats are most likely fairly similar across shards, so I don't think we're going to run into problems. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:41pm UTC](https://discuss.elastic.co/t/checking-for-sparse-statistics-and-problems-with-shard-routing/33227/4 "2017-07-05T23:41:47Z")
Closed on July 5, 2017, 11:41pm UTC