# Advice needed on ES indexes/structure **URL:** \ **Category:** Elasticsearch\ **Created:** [June 14, 2013, 3:15pm UTC](https://discuss.elastic.co/t/advice-needed-on-es-indexes-structure/12419 "2013-06-14T15:15:43Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Mark\_van\_der\_Velden](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_van_der_velden/32/2298_2.png) [@Mark\_van\_der\_Velden](https://discuss.elastic.co/u/Mark_van_der_Velden)\ **Post date:** [June 14, 2013, 3:15pm UTC](https://discuss.elastic.co/t/advice-needed-on-es-indexes-structure/12419/1 "2013-06-14T15:15:43Z")
Hi, In a project we'll have data from various sources. A lot of these sources can be divided as such: All data belongs to a certain group (1:m) and there can be multiple groups. The data in a group is always unrelated to the data in another group. It's hard to say how much the ratio will be, but initial guesses guestimate that the ratio will be more than 1000:1 (1000 "data" compared to 1 "group"). Data can be seen as chunks of text (of about 200 characters) and meta-data. Within a group there can be various types of data. It can be file meta-data, plain-text or "structured" JSON. My questions are: - Is it a good idea to create an index for each group, or would a better approach be to create an index for the various types? - How does this work with sharding or is that an unrelated issue? - Does it have any (availability/performance) benefit to keep data contained within an index because it's related (since there is no shared data between groups) The document refers to an index as a schema, as such it would hint to making an index per type. Id just like to know if that would be a correct assumption, as there is a valid case for both options. Thanks in advance, Mark -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [June 16, 2013, 12:17pm UTC](https://discuss.elastic.co/t/advice-needed-on-es-indexes-structure/12419/2 "2013-06-16T12:17:27Z")
Hello Mark, From a functionality point of view, each type has a mapping attached to it, which acts as a schema. On the other hand, all your types end up in the same Lucene indices (shards), which means all fields from all types end up in the same "master" schema, and the type itself is just another field. Having one index per group has the benefit that you only have that index to search on, but you'll have a lot of indices, and implicitly a lot of shards. Which comes with an overhead in terms of memory and open files. On the other hand, having all data into the same index would normally require searching in all shards to find data for a single group. You can work around that by using routing [http://www.elasticsearch.org/guide/reference/mapping/routing-field/](http://www.elasticsearch.org/guide/reference/mapping/routing-field/) Since you have a low number of documents per group, you'll probably be better off having them all in the same index and using routing. The problem comes if you have groups with A LOT of documents, because with routing all documents from a group will all end up in the same shard - which might make your shard sizes different. Then, you'd probably have to take those big groups out in their own indices, and use aliases[http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases/](http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases/)to trick apps into thinking it's a single index when they search 🙂 I'd recommend two talks about what you're interested in: - this one[http://www.elasticsearch.org/videos/scaling-massive-elasticsearch-clusters/](http://www.elasticsearch.org/videos/scaling-massive-elasticsearch-clusters/)is about routing and caching and other stuff - this one[http://www.elasticsearch.org/videos/big-data-search-and-analytics/](http://www.elasticsearch.org/videos/big-data-search-and-analytics/)is about design patterns (like having one index vs multiple indices) ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene On Fri, Jun 14, 2013 at 6:15 PM, Mark van der Velden [markvdv@gmail.com](mailto:markvdv@gmail.com)wrote: > Hi, > > In a project we'll have data from various sources. A lot of these sources > can be divided as such: > > All data belongs to a certain group (1:m) and there can be multiple > groups. The data in a group is always unrelated to the data in another > group. It's hard to say how much the ratio will be, but initial guesses > guestimate that the ratio will be more than 1000:1 (1000 "data" compared to > 1 "group"). Data can be seen as chunks of text (of about 200 characters) > and meta-data. Within a group there can be various types of data. It can be > file meta-data, plain-text or "structured" JSON. > > My questions are: > > - Is it a good idea to create an index for each group, or would a better > approach be to create an index for the various types? > - How does this work with sharding or is that an unrelated issue? > - Does it have any (availability/performance) benefit to keep data > contained within an index because it's related (since there is no shared > data between groups) > > The document refers to an index as a schema, as such it would hint to > making an index per type. Id just like to know if that would be a correct > assumption, as there is a valid case for both options. > > Thanks in advance, > > Mark > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mark\_van\_der\_Velden](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_van_der_velden/32/2298_2.png) [@Mark\_van\_der\_Velden](https://discuss.elastic.co/u/Mark_van_der_Velden)\ **Post date:** [June 17, 2013, 7:23am UTC](https://discuss.elastic.co/t/advice-needed-on-es-indexes-structure/12419/3 "2013-06-17T07:23:26Z")
Hi Radu, First of all, thanks for your feedback. On Sunday, 16 June 2013 14:17:27 UTC+2, Radu Gheorghe wrote: > Hello Mark, > > From a functionality point of view, each type has a mapping attached to > it, which acts as a schema. On the other hand, all your types end up in the > same Lucene indices (shards), which means all fields from all types end up > in the same "master" schema, and the type itself is just another field. > > Having one index per group has the benefit that you only have that index > to search on, but you'll have a lot of indices, and implicitly a lot of > shards. Which comes with an overhead in terms of memory and open files. > > On the other hand, having all data into the same index would normally > require searching in all shards to find data for a single group. You can > work around that by using routing > [http://www.elasticsearch.org/guide/reference/mapping/routing-field/](http://www.elasticsearch.org/guide/reference/mapping/routing-field/) > > Since you have a low number of documents per group, you'll probably be > better off having them all in the same index and using routing. The problem > comes if you have groups with A LOT of documents, because with routing all > documents from a group will all end up in the same shard - which might make > your shard sizes different. Then, you'd probably have to take those big > groups out in their own indices, and use aliases[http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases/](http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases/)to trick apps into thinking it's a single index when they search 🙂 To give a little context about numbers I'm considering: - Some groups with around 100.000 documents, per year - Many groups with around 1.000.000 documents, per year. - A reasonable amount of groups having 150.000.000 documents per year. I estimate having an increase in groups of about 500.000, each year. > I'd recommend two talks about what you're interested in: > > - this one[http://www.elasticsearch.org/videos/scaling-massive-elasticsearch-clusters/](http://www.elasticsearch.org/videos/scaling-massive-elasticsearch-clusters/)is about routing and caching and other stuff > - this one[http://www.elasticsearch.org/videos/big-data-search-and-analytics/](http://www.elasticsearch.org/videos/big-data-search-and-analytics/)is about design patterns (like having one index vs multiple indices) Great, I'll watch these right away. > ## Best regards, Radu > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > On Fri, Jun 14, 2013 at 6:15 PM, Mark van der Velden \<[mar...@gmail.com](mailto:mar...@gmail.com)\ > > > wrote: > > > Hi, > > > > In a project we'll have data from various sources. A lot of these sources > > can be divided as such: > > > > All data belongs to a certain group (1:m) and there can be multiple > > groups. The data in a group is always unrelated to the data in another > > group. It's hard to say how much the ratio will be, but initial guesses > > guestimate that the ratio will be more than 1000:1 (1000 "data" compared to > > 1 "group"). Data can be seen as chunks of text (of about 200 characters) > > and meta-data. Within a group there can be various types of data. It can be > > file meta-data, plain-text or "structured" JSON. > > > > My questions are: > > > > - Is it a good idea to create an index for each group, or would a better > > approach be to create an index for the various types? > > - How does this work with sharding or is that an unrelated issue? > > - Does it have any (availability/performance) benefit to keep data > > contained within an index because it's related (since there is no shared > > data between groups) > > > > The document refers to an index as a schema, as such it would hint to > > making an index per type. Id just like to know if that would be a correct > > assumption, as there is a valid case for both options. > > > > Thanks in advance, > > > > Mark > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:30am UTC](https://discuss.elastic.co/t/advice-needed-on-es-indexes-structure/12419/4 "2017-07-06T02:30:52Z")