# Multi billion documents index **URL:** \ **Category:** Elasticsearch\ **Created:** [November 12, 2015, 12:30am UTC](https://discuss.elastic.co/t/multi-billion-documents-index/34368 "2015-11-12T00:30:09Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Adam\_Smith1](https://avatars.discourse-cdn.com/v4/letter/a/90db22/32.png) [@Adam\_Smith1](https://discuss.elastic.co/u/Adam_Smith1)\ **Post date:** [November 12, 2015, 12:30am UTC](https://discuss.elastic.co/t/multi-billion-documents-index/34368/1 "2015-11-12T00:30:09Z")
Dear community, I want to use elasticsearch to index inlinks of websites, currently 2 Billion Websites and ~ 500 Billion links, but only 100 Million domains. My queries shall only focus inlinks by domain - is there any chance to exploit this natural partitioning? I want to index fulltext of anchor-links and some integer and double properties, say in average 100 bytes per document. Some domains like Google can have up to billions of inlinks, other a very low number. I could maybe use a large number of shards to adress the 2B border, but I fear the non-linear scaling behaviour of searches when having all 500B inlinks in one index. And I have no idea how indexing would behave for that number of documents. I was also thinking to use something like [https://code.google.com/p/lucene-on-cassandra/](https://code.google.com/p/lucene-on-cassandra/) where I could create a separate index per domain. But I would really like to use ES, certainly. Would ES filters lead to the same behaviour when working on scale? Means that only links/documents from a domain are considered and hence response times can be reduced? Is there any other solution to achieve my goals? Thanks for any advice! Adam ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 12, 2015, 3:39am UTC](https://discuss.elastic.co/t/multi-billion-documents-index/34368/2 "2015-11-12T03:39:29Z")
An index per domain would mean a lot of indices, and you may run into [this](https://www.elastic.co/guide/en/elasticsearch/guide/current/kagillion-shards.html). Maybe look at an index per country code/TLD, and use routing? ---
**Author:** ![mainec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mainec/32/5557_2.png) [@mainec](https://discuss.elastic.co/u/mainec)\ **Post date:** [November 12, 2015, 11:31am UTC](https://discuss.elastic.co/t/multi-billion-documents-index/34368/3 "2015-11-12T11:31:55Z")
> [@Adam\_Smith1](#): > > but I fear the non-linear scaling behaviour of searches when having all 500B inlinks in one index Your problem sounds very much like [One Big User | Elasticsearch: The Definitive Guide [2.x] | Elastic](https://www.elastic.co/guide/en/elasticsearch/guide/current/one-big-user.html) to me. Though not focussed on web crawling maybe the post helps you anyhow. Isabel ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:38pm UTC](https://discuss.elastic.co/t/multi-billion-documents-index/34368/4 "2017-07-05T23:38:59Z")