# Mapping Size limitations **URL:** \ **Category:** Elasticsearch\ **Created:** [November 10, 2012, 10:12pm UTC](https://discuss.elastic.co/t/mapping-size-limitations/9660 "2012-11-10T22:12:51Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![revdev](https://avatars.discourse-cdn.com/v4/letter/r/ecb155/32.png) [@revdev](https://discuss.elastic.co/u/revdev)\ **Post date:** [November 10, 2012, 10:12pm UTC](https://discuss.elastic.co/t/mapping-size-limitations/9660/1 "2012-11-10T22:12:51Z")
Hi, I am using dynamic templates and I am dealing with with couple thousand such dynamic fields. Each of these dynamic fields are objects have 2 to 3 subfields which are of type "byte". My question is , is there any performance penalty of having a large mapping? Right now I have couple thousand fields in mapping but in future it can increase to maybe 10k-20k fields. Will I see performance degradation with large mapping file. If so, what things will be effected? FYI, I am planning to use facets over these fields. Thanks! -- ---
**Author:** ![Chris\_Male](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_male/32/2607_2.png) [@Chris\_Male](https://discuss.elastic.co/u/Chris_Male)\ **Post date:** [November 12, 2012, 2:52am UTC](https://discuss.elastic.co/t/mapping-size-limitations/9660/2 "2012-11-12T02:52:08Z")
Hi, The first thing that jumps out at me is, do you really need that many fields? Are you able to give us a bit of information about what you're doing and maybe we can come up with a way to do it without 20,000 fields. On Sunday, November 11, 2012 11:12:51 AM UTC+13, revdev wrote: > Hi, > > I am using dynamic templates and I am dealing with with couple thousand > such dynamic fields. Each of these dynamic fields are objects have 2 to 3 > subfields which are of type "byte". My question is , is there any > performance penalty of having a large mapping? Right now I have couple > thousand fields in mapping but in future it can increase to maybe 10k-20k > fields. Will I see performance degradation with large mapping file. If so, > what things will be effected? FYI, I am planning to use facets over these > fields. > > Thanks! -- ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 12, 2012, 8:32am UTC](https://discuss.elastic.co/t/mapping-size-limitations/9660/3 "2012-11-12T08:32:50Z")
It's nearly impossible to manage 20k fields. The reasons are: each field consumes around some MB of resident memory in Lucene, each field mapping creation causes cluster blocks and proliferation of the mapping settings throughout the cluster nodes. Even if you manage to get 20k field created, constructing a query over 20k fields and the lookup time for each field settings and the mappings will eat up your performance. Rule of thumbs: Facets are designed to perform well over a small number of fields with a high cardinality of values. They do not perform well over a high number of fields with low cardinality. I would also be curious to learn about the scenario in which such a high number of fields is required. Jörg On Saturday, November 10, 2012 11:12:51 PM UTC+1, revdev wrote: > Hi, > > I am using dynamic templates and I am dealing with with couple thousand > such dynamic fields. Each of these dynamic fields are objects have 2 to 3 > subfields which are of type "byte". My question is , is there any > performance penalty of having a large mapping? Right now I have couple > thousand fields in mapping but in future it can increase to maybe 10k-20k > fields. Will I see performance degradation with large mapping file. If so, > what things will be effected? FYI, I am planning to use facets over these > fields. > > Thanks! -- ---
**Author:** ![Vinay\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinay_2/32/2344_2.png) [@Vinay\_2](https://discuss.elastic.co/u/Vinay_2)\ **Post date:** [November 12, 2012, 4:47pm UTC](https://discuss.elastic.co/t/mapping-size-limitations/9660/4 "2012-11-12T16:47:20Z")
Thanks Jorg, Chris for quick reply. Let me explain you the situation which will clear what I am trying to accomplish. I am not giving the exact domain example but a similar example in different domain. Scenario: Lets say, I am storing a list of restaurant reviews from all over the web. Each review document can have following fields: review\_id (long) review\_ratings (object) aspect\_1\_name (string) : rating (float) aspect\_2\_name (string) : rating (float) aspect\_3\_name (string) : rating (float) ... date (datetime) Requirement: The goal is to be able to calculate facets on "review aspects" and get avg rating given for each aspect across all reviews within a given period. In this case, aspects can be like "Visual appeal of dish", "taste", "smell" etc. Hypothetically, lets assume number of aspects can increase to 20k aspects. Note that, a single review can have maybe a dozen of aspects defined but when you talk about millions of reviews over a period, they might have collectively thousands of different aspects associated. So, here our mapping will become huge because of "review\_ratings" object. Now, to achieve this, I can use histogram facet over Field Date and Value as the Aspect Name. To get facets over, say 100 aspects, I can create 100 facets, one for each aspect. So, at a time, I will just be querying around 100 aspects at a time to get their avg rating. Now that you know a sample scenario, can you guys tell me if my approach is correct? or I am doing something fundamentally wrong ? Thanks a lot again for help guys! Vinay On Mon, Nov 12, 2012 at 12:32 AM, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > It's nearly impossible to manage 20k fields. The reasons are: each field > consumes around some MB of resident memory in Lucene, each field mapping > creation causes cluster blocks and proliferation of the mapping settings > throughout the cluster nodes. Even if you manage to get 20k field created, > constructing a query over 20k fields and the lookup time for each field > settings and the mappings will eat up your performance. > > Rule of thumbs: Facets are designed to perform well over a small number of > fields with a high cardinality of values. They do not perform well over a > high number of fields with low cardinality. > > I would also be curious to learn about the scenario in which such a high > number of fields is required. > > Jörg > > On Saturday, November 10, 2012 11:12:51 PM UTC+1, revdev wrote: > > > Hi, > > > > I am using dynamic templates and I am dealing with with couple thousand > > such dynamic fields. Each of these dynamic fields are objects have 2 to 3 > > subfields which are of type "byte". My question is , is there any > > performance penalty of having a large mapping? Right now I have couple > > thousand fields in mapping but in future it can increase to maybe 10k-20k > > fields. Will I see performance degradation with large mapping file. If so, > > what things will be effected? FYI, I am planning to use facets over these > > fields. > > > > Thanks! > > -- -- ---
**Author:** ![es\_learner](https://avatars.discourse-cdn.com/v4/letter/e/8dc957/32.png) [@es\_learner](https://discuss.elastic.co/u/es_learner)\ **Post date:** [November 12, 2012, 10:18pm UTC](https://discuss.elastic.co/t/mapping-size-limitations/9660/5 "2012-11-12T22:18:07Z")
Do you already have some histogram data exhibiting the 20k spread? I can see the possibility of 20k aspects and I can also see a very long tail. Can the aspects be sub-grouped into doc\_type's thereby reducing the set of aspects for each sub-group? e.g. 'jasmine flavor' won't apply to fries/steaks but fine with tea. ---
**Author:** ![revdev](https://avatars.discourse-cdn.com/v4/letter/r/ecb155/32.png) [@revdev](https://discuss.elastic.co/u/revdev)\ **Post date:** [November 12, 2012, 10:58pm UTC](https://discuss.elastic.co/t/mapping-size-limitations/9660/6 "2012-11-12T22:58:43Z")
You mean, create a separate Index Type for each subgroups? I can do that but that's the last resort I want to take since it will require quite a lot of code to make things look seamless to outside clients for both query and indexing. Thanks for pitching that idea tho. On Monday, November 12, 2012 2:18:13 PM UTC-8, es\_learner wrote: > Do you already have some histogram data exhibiting the 20k spread? I can > see > the possibility of 20k aspects and I can also see a very long tail. Can > the > aspects be sub-grouped into doc\_type's thereby reducing the set of aspects > for each sub-group? e.g. 'jasmine flavor' won't apply to fries/steaks but > fine with tea. > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Mapping-Size-limitations-tp4025257p4025334.html](http://elasticsearch-users.115913.n3.nabble.com/Mapping-Size-limitations-tp4025257p4025334.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). -- ---
**Author:** ![Chris\_Male](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_male/32/2607_2.png) [@Chris\_Male](https://discuss.elastic.co/u/Chris_Male)\ **Post date:** [November 12, 2012, 11:13pm UTC](https://discuss.elastic.co/t/mapping-size-limitations/9660/7 "2012-11-12T23:13:23Z")
Practically I can't see how you're going to be able to support so many fields. Jorg is right, memory consumption is going to be immense, things will get unmanageable. What if, along the lines of what es\_learner suggested, you have a document per aspect? So each document could consist of review id, aspect\_name, rating and datetime. You could then filter aspect\_name to choose which aspects you wanted to facet on. On Tuesday, November 13, 2012 11:58:44 AM UTC+13, revdev wrote: > You mean, create a separate Index Type for each subgroups? I can do that > but that's the last resort I want to take since it will require quite a lot > of code to make things look seamless to outside clients for both query and > indexing. > > Thanks for pitching that idea tho. > > On Monday, November 12, 2012 2:18:13 PM UTC-8, es\_learner wrote: > > > Do you already have some histogram data exhibiting the 20k spread? I can > > see > > the possibility of 20k aspects and I can also see a very long tail. Can > > the > > aspects be sub-grouped into doc\_type's thereby reducing the set of > > aspects > > for each sub-group? e.g. 'jasmine flavor' won't apply to fries/steaks > > but > > fine with tea. > > > > -- > > View this message in context: > > [http://elasticsearch-users.115913.n3.nabble.com/Mapping-Size-limitations-tp4025257p4025334.html](http://elasticsearch-users.115913.n3.nabble.com/Mapping-Size-limitations-tp4025257p4025334.html) > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). -- ---
**Author:** ![revdev](https://avatars.discourse-cdn.com/v4/letter/r/ecb155/32.png) [@revdev](https://discuss.elastic.co/u/revdev)\ **Post date:** [November 12, 2012, 11:28pm UTC](https://discuss.elastic.co/t/mapping-size-limitations/9660/8 "2012-11-12T23:28:23Z")
hmm, let me tell you one thing I tried but it did not work: I changed my doc from this review\_id (long) review\_ratings (_object_) aspect\_1\_name (string) : rating (float) aspect\_2\_name (string) : rating (float) aspect\_3\_name (string) : rating (float) ... date (datetime) to review\_id (long) review\_ratings (_Array_) [ {"name" : aspect\_1\_name (string), "rating": aspect\_rating (float) }, {"name" : aspect\_2\_name (string), "rating": aspect\_rating (float) }, ... ] date (datetime) Then I applied facet filter to filter aspects by their name, ie. "facet\_filter": { 'term' : { "name" : "aspect\_1\_name"}} but the result was returning result by calculating mean on all elements within "review\_ratings" array for reviews which had "aspect\_name\_1" in the review\_rating array. For example take this example rating\_array: { "rating\_array" : [ {"name" : "a", "rating" : 3}, {"name" : "b", "rating" : 4}, {"name" : "c", "rating" : 5}, ] } If I filtering by "name":b", ES will calculate total as 12 and mean as 12/3, rather than just returning total as 4 and mean as 4. Can I do something that will just return results by calculating just the data from the array index which has name="aspect\_1\_name" ? or, am I doing something wrong? 🙂 thx! On Monday, November 12, 2012 3:13:23 PM UTC-8, Chris Male wrote: > Practically I can't see how you're going to be able to support so many > fields. Jorg is right, memory consumption is going to be immense, things > will get unmanageable. > > What if, along the lines of what es\_learner suggested, you have a document > per aspect? So each document could consist of review id, aspect\_name, > rating and datetime. You could then filter aspect\_name to choose which > aspects you wanted to facet on. > > On Tuesday, November 13, 2012 11:58:44 AM UTC+13, revdev wrote: > > > You mean, create a separate Index Type for each subgroups? I can do that > > but that's the last resort I want to take since it will require quite a lot > > of code to make things look seamless to outside clients for both query and > > indexing. > > > > Thanks for pitching that idea tho. > > > > On Monday, November 12, 2012 2:18:13 PM UTC-8, es\_learner wrote: > > > > > Do you already have some histogram data exhibiting the 20k spread? I > > > can see > > > the possibility of 20k aspects and I can also see a very long tail. Can > > > the > > > aspects be sub-grouped into doc\_type's thereby reducing the set of > > > aspects > > > for each sub-group? e.g. 'jasmine flavor' won't apply to fries/steaks > > > but > > > fine with tea. > > > > > > -- > > > View this message in context: > > > [http://elasticsearch-users.115913.n3.nabble.com/Mapping-Size-limitations-tp4025257p4025334.html](http://elasticsearch-users.115913.n3.nabble.com/Mapping-Size-limitations-tp4025257p4025334.html) > > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). -- ---
**Author:** ![es\_learner](https://avatars.discourse-cdn.com/v4/letter/e/8dc957/32.png) [@es\_learner](https://discuss.elastic.co/u/es_learner)\ **Post date:** [November 12, 2012, 11:38pm UTC](https://discuss.elastic.co/t/mapping-size-limitations/9660/9 "2012-11-12T23:38:52Z")
Is your (new) Array of review\_ratings a nested type? [http://www.elasticsearch.org/guide/reference/mapping/nested-type.html](http://www.elasticsearch.org/guide/reference/mapping/nested-type.html) ---
**Author:** ![revdev](https://avatars.discourse-cdn.com/v4/letter/r/ecb155/32.png) [@revdev](https://discuss.elastic.co/u/revdev)\ **Post date:** [November 12, 2012, 11:41pm UTC](https://discuss.elastic.co/t/mapping-size-limitations/9660/10 "2012-11-12T23:41:03Z")
not now but that might be the solution I was looking for! 🙂 Let me experiment with nested\_types. Thanks a lot es\_learner! -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:04am UTC](https://discuss.elastic.co/t/mapping-size-limitations/9660/11 "2017-07-06T03:04:39Z")