# Architecture question **URL:** \ **Category:** Elasticsearch\ **Created:** [September 25, 2013, 3:42pm UTC](https://discuss.elastic.co/t/architecture-question/13754 "2013-09-25T15:42:02Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![fred](https://avatars.discourse-cdn.com/v4/letter/f/db5fbb/32.png) [@fred](https://discuss.elastic.co/u/fred)\ **Post date:** [September 25, 2013, 3:42pm UTC](https://discuss.elastic.co/t/architecture-question/13754/1 "2013-09-25T15:42:02Z")
Good morning (ugt), I'm asking myself an important architecture question, and since it's still time to change our software architecture, I'd like your opinion on how our ES cluster will handle this and what's the best way to do it. We have clients, that have many "crawl" objects associated, and each crawl has hundreds thousands records (expects hundreds of gigabytes quickly) We're currently using ES this way: - each client has an index - on each index, we create one crawl\_XXX mapping (where XXX is the number associated to the crawl object). In each crawl\_XXX, we have the records. Pros: it's easy to find somethign. Cons: hundreds of thousands indexes with thousand mappings, all the same. I was wondering if it would not be better to use: - a single index - with client\_id - with crawl\_id Any constructive opinion appreciated. Thank you very much -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Graham\_Lenton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/graham_lenton/32/860_2.png) [@Graham\_Lenton](https://discuss.elastic.co/u/Graham_Lenton)\ **Post date:** [September 26, 2013, 7:32am UTC](https://discuss.elastic.co/t/architecture-question/13754/2 "2013-09-26T07:32:02Z")
Hundreds of thousands of indices is definitely NOT a good idea. Even if every index had only a single shard, and no replicas, then your system would need to maintain hundreds of thousands of open file handles. Is is possible to partition your data by timestamp ? e.g. have an index for each day / week / month / year ? On Wednesday, 25 September 2013 15:42:02 UTC, [fr...@botify.com](mailto:fr...@botify.com) wrote: > Good morning (ugt), > > I'm asking myself an important architecture question, and since it's still > time to change our software architecture, I'd like your opinion on how our > ES cluster will handle this and what's the best way to do it. > > We have clients, that have many "crawl" objects associated, and each crawl > has hundreds thousands records (expects hundreds of gigabytes quickly) > > We're currently using ES this way: > > - each client has an index > - on each index, we create one crawl\_XXX mapping (where XXX is the number > associated to the crawl object). In each crawl\_XXX, we have the records. > > Pros: it's easy to find somethign. > Cons: hundreds of thousands indexes with thousand mappings, all the same. > > I was wondering if it would not be better to use: > > - a single index > - with client\_id > - with crawl\_id > > Any constructive opinion appreciated. > Thank you very much -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 26, 2013, 7:39am UTC](https://discuss.elastic.co/t/architecture-question/13754/3 "2013-09-26T07:39:40Z")
The crawled docs seem to be of the same structure, and you have no requirements of multi-tenancy, so there is absolutely no need to use more than one index. So why don't you use a timestamp-rolling index with a global unique source ID (e.g. URL or client ID + timestamp) as doc ID? Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:14am UTC](https://discuss.elastic.co/t/architecture-question/13754/4 "2017-07-06T02:14:37Z")
Closed on July 6, 2017, 2:14am UTC