# Is it OK to have a large number of sparsely populated fields?
**URL:** https://discuss.elastic.co/t/is-it-ok-to-have-a-large-number-of-sparsely-populated-fields/4563
**Category:** Elasticsearch
**Created:** [June 7, 2011, 3:18am UTC](https://discuss.elastic.co/t/is-it-ok-to-have-a-large-number-of-sparsely-populated-fields/4563 "2011-06-07T03:18:23Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Andy\_2](https://discuss.elastic.co/u/Andy_2)
#### Post date: [June 7, 2011, 3:18am UTC](https://discuss.elastic.co/t/is-it-ok-to-have-a-large-number-of-sparsely-populated-fields/4563/1 "2011-06-07T03:18:23Z")
I have many categories of products. Each category has its own unique
fields.
For example, category "Apparel" has fields "size", "color", "style".
Category "laptop" has fields "processor", "RAM", "screen size". etc
etc.
So if I use 1 index for all the products, there will be many
(potentially tens of thousands) fields. Each field will only be used
by a small portion of products. Is this "1 index" approach a good
design? Will all those empty fields lead to wasted space?
Thanks
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [June 7, 2011, 5:18am UTC](https://discuss.elastic.co/t/is-it-ok-to-have-a-large-number-of-sparsely-populated-fields/4563/2 "2011-06-07T05:18:40Z")
Less of a wasted space, but, a field does come with a memory overhead.
On Tuesday, June 7, 2011 at 6:18 AM, Andy wrote:
> I have many categories of products. Each category has its own unique
> fields.
>
> For example, category "Apparel" has fields "size", "color", "style".
> Category "laptop" has fields "processor", "RAM", "screen size". etc
> etc.
>
> So if I use 1 index for all the products, there will be many
> (potentially tens of thousands) fields. Each field will only be used
> by a small portion of products. Is this "1 index" approach a good
> design? Will all those empty fields lead to wasted space?
>
> Thanks
---
### Author:  [@Andy\_2](https://discuss.elastic.co/u/Andy_2)
#### Post date: [June 7, 2011, 5:58pm UTC](https://discuss.elastic.co/t/is-it-ok-to-have-a-large-number-of-sparsely-populated-fields/4563/3 "2011-06-07T17:58:37Z")
Can you explain a bit on where the memory overhead comes from?
What kind of index structure would you recommend in this case?
Thanks.
On Jun 7, 1:18 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> Less of a wasted space, but, a field does come with a memory overhead.
>
> On Tuesday, June 7, 2011 at 6:18 AM, Andy wrote:
>
> > I have many categories of products. Each category has its own unique
> > fields.
>
> > For example, category "Apparel" has fields "size", "color", "style".
> > Category "laptop" has fields "processor", "RAM", "screen size". etc
> > etc.
>
> > So if I use 1 index for all the products, there will be many
> > (potentially tens of thousands) fields. Each field will only be used
> > by a small portion of products. Is this "1 index" approach a good
> > design? Will all those empty fields lead to wasted space?
>
> > Thanks
---
### Author:  [@Berkay\_Mollamustafao](https://discuss.elastic.co/u/Berkay_Mollamustafao)
#### Post date: [June 7, 2011, 6:05pm UTC](https://discuss.elastic.co/t/is-it-ok-to-have-a-large-number-of-sparsely-populated-fields/4563/4 "2011-06-07T18:05:07Z")
Couple of things if I understood correctly.
1. You don't have to store the docs with separate fields. You can store it
as json and data would be searchable.
2. You can use different "types" in a single index. The docs at each type
can have different fields.
In short, depending on number of documents you have, you may be able to
store all your data in a single index with multiple types.
Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype
On Tue, Jun 7, 2011 at 1:58 PM, Andy [selforganized@gmail.com](mailto:selforganized@gmail.com) wrote:
> Can you explain a bit on where the memory overhead comes from?
>
> What kind of index structure would you recommend in this case?
>
> Thanks.
>
> On Jun 7, 1:18 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
>
> > Less of a wasted space, but, a field does come with a memory overhead.
> >
> > On Tuesday, June 7, 2011 at 6:18 AM, Andy wrote:
> >
> > > I have many categories of products. Each category has its own unique
> > > fields.
> >
> > > For example, category "Apparel" has fields "size", "color", "style".
> > > Category "laptop" has fields "processor", "RAM", "screen size". etc
> > > etc.
> >
> > > So if I use 1 index for all the products, there will be many
> > > (potentially tens of thousands) fields. Each field will only be used
> > > by a small portion of products. Is this "1 index" approach a good
> > > design? Will all those empty fields lead to wasted space?
> >
> > > Thanks
---
### Author:  [@jacorob](https://discuss.elastic.co/u/jacorob)
#### Post date: [June 7, 2011, 7:28pm UTC](https://discuss.elastic.co/t/is-it-ok-to-have-a-large-number-of-sparsely-populated-fields/4563/5 "2011-06-07T19:28:18Z")
You may be aware of this, but since it wasn't obvious to me when I first
started out...
> 1. You can use different "types" in a single index. The docs at each
> type can have different fields.
Just be aware that when using a single index the fields across "types"
are not 100% independent of each other. By this I mean that if two different
"types"
have a field with the same name, the mapping type (number, boolean, etc)
of BOTH fields MUST be the same in both types.
Or as Shay explained in more detail (and probably more concisely) in a forum
post:
[http://elasticsearch-users.115913.n3.nabble.com/Searching-across-types-tp1745420p1749010.html](http://elasticsearch-users.115913.n3.nabble.com/Searching-across-types-tp1745420p1749010.html)
Bob
On Tue, Jun 7, 2011 at 1:05 PM, Berkay Mollamustafaoglu
[mberkay@gmail.com](mailto:mberkay@gmail.com)wrote:
> Couple of things if I understood correctly.
>
> 1. You don't have to store the docs with separate fields. You can store it
> as json and data would be searchable.
> 2. You can use different "types" in a single index. The docs at each type
> can have different fields.
> In short, depending on number of documents you have, you may be able to
> store all your data in a single index with multiple types.
>
> Regards,
> Berkay Mollamustafaoglu
> mberkay on yahoo, google and skype
>
> On Tue, Jun 7, 2011 at 1:58 PM, Andy [selforganized@gmail.com](mailto:selforganized@gmail.com) wrote:
>
> > Can you explain a bit on where the memory overhead comes from?
> >
> > What kind of index structure would you recommend in this case?
> >
> > Thanks.
> >
> > On Jun 7, 1:18 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> >
> > > Less of a wasted space, but, a field does come with a memory overhead.
> > >
> > > On Tuesday, June 7, 2011 at 6:18 AM, Andy wrote:
> > >
> > > > I have many categories of products. Each category has its own unique
> > > > fields.
> > >
> > > > For example, category "Apparel" has fields "size", "color", "style".
> > > > Category "laptop" has fields "processor", "RAM", "screen size". etc
> > > > etc.
> > >
> > > > So if I use 1 index for all the products, there will be many
> > > > (potentially tens of thousands) fields. Each field will only be used
> > > > by a small portion of products. Is this "1 index" approach a good
> > > > design? Will all those empty fields lead to wasted space?
> > >
> > > > Thanks
---
### Author:  [@Andy\_2](https://discuss.elastic.co/u/Andy_2)
#### Post date: [June 7, 2011, 7:29pm UTC](https://discuss.elastic.co/t/is-it-ok-to-have-a-large-number-of-sparsely-populated-fields/4563/6 "2011-06-07T19:29:54Z")
If I store the data as json without separate fields, will I still be
able to facet on the fields?
For example, when a user is searching on "apparel", I need to be able
to facet on fields like "size" and "color".
Thanks.
On Jun 7, 2:05 pm, Berkay Mollamustafaoglu [mber...@gmail.com](mailto:mber...@gmail.com) wrote:
> Couple of things if I understood correctly.
>
> 1. You don't have to store the docs with separate fields. You can store it
> as json and data would be searchable.
> 2. You can use different "types" in a single index. The docs at each type
> can have different fields.
> In short, depending on number of documents you have, you may be able to
> store all your data in a single index with multiple types.
>
> Regards,
> Berkay Mollamustafaoglu
> mberkay on yahoo, google and skype
>
> On Tue, Jun 7, 2011 at 1:58 PM, Andy [selforgani...@gmail.com](mailto:selforgani...@gmail.com) wrote:
>
> > Can you explain a bit on where the memory overhead comes from?
>
> > What kind of index structure would you recommend in this case?
>
> > Thanks.
>
> > On Jun 7, 1:18 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote:
> >
> > > Less of a wasted space, but, a field does come with a memory overhead.
>
> > > On Tuesday, June 7, 2011 at 6:18 AM, Andy wrote:
> > >
> > > > I have many categories of products. Each category has its own unique
> > > > fields.
>
> > > > For example, category "Apparel" has fields "size", "color", "style".
> > > > Category "laptop" has fields "processor", "RAM", "screen size". etc
> > > > etc.
>
> > > > So if I use 1 index for all the products, there will be many
> > > > (potentially tens of thousands) fields. Each field will only be used
> > > > by a small portion of products. Is this "1 index" approach a good
> > > > design? Will all those empty fields lead to wasted space?
>
> > > > Thanks
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [June 8, 2011, 9:26am UTC](https://discuss.elastic.co/t/is-it-ok-to-have-a-large-number-of-sparsely-populated-fields/4563/7 "2011-06-08T09:26:31Z")
No, you will need to store them on different fields. The memory associated with each field index is an in memory data structure stored in the search engine to make it searchable.
On Tuesday, June 7, 2011 at 10:29 PM, Andy wrote:
> If I store the data as json without separate fields, will I still be
> able to facet on the fields?
>
> For example, when a user is searching on "apparel", I need to be able
> to facet on fields like "size" and "color".
>
> Thanks.
>
> On Jun 7, 2:05 pm, Berkay Mollamustafaoglu \<[mber...@gmail.com](mailto:mber...@gmail.com) ([http://gmail.com](http://gmail.com))\> wrote:
>
> > Couple of things if I understood correctly.
> >
> > 1. You don't have to store the docs with separate fields. You can store it
> > as json and data would be searchable.
> > 2. You can use different "types" in a single index. The docs at each type
> > can have different fields.
> > In short, depending on number of documents you have, you may be able to
> > store all your data in a single index with multiple types.
> >
> > Regards,
> > Berkay Mollamustafaoglu
> > mberkay on yahoo, google and skype
> >
> > On Tue, Jun 7, 2011 at 1:58 PM, Andy \<[selforgani...@gmail.com](mailto:selforgani...@gmail.com) ([http://gmail.com](http://gmail.com))\> wrote:
> >
> > > Can you explain a bit on where the memory overhead comes from?
> >
> > > What kind of index structure would you recommend in this case?
> >
> > > Thanks.
> >
> > > On Jun 7, 1:18 am, Shay Banon \<[shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) ([http://elasticsearch.com](http://elasticsearch.com))\> wrote:
> > >
> > > > Less of a wasted space, but, a field does come with a memory overhead.
> >
> > > > On Tuesday, June 7, 2011 at 6:18 AM, Andy wrote:
> > > >
> > > > > I have many categories of products. Each category has its own unique
> > > > > fields.
> >
> > > > > For example, category "Apparel" has fields "size", "color", "style".
> > > > > Category "laptop" has fields "processor", "RAM", "screen size". etc
> > > > > etc.
> >
> > > > > So if I use 1 index for all the products, there will be many
> > > > > (potentially tens of thousands) fields. Each field will only be used
> > > > > by a small portion of products. Is this "1 index" approach a good
> > > > > design? Will all those empty fields lead to wasted space?
> >
> > > > > Thanks
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:04am UTC](https://discuss.elastic.co/t/is-it-ok-to-have-a-large-number-of-sparsely-populated-fields/4563/8 "2017-07-06T04:04:25Z")