# Nested object with arrays , best practice
**URL:** https://discuss.elastic.co/t/nested-object-with-arrays-best-practice/9378
**Category:** Elasticsearch
**Created:** [October 16, 2012, 12:31pm UTC](https://discuss.elastic.co/t/nested-object-with-arrays-best-practice/9378 "2012-10-16T12:31:06Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@samuel\_merlet](https://discuss.elastic.co/u/samuel_merlet)
#### Post date: [October 16, 2012, 12:31pm UTC](https://discuss.elastic.co/t/nested-object-with-arrays-best-practice/9378/1 "2012-10-16T12:31:06Z")
Hi,
I have a quite complex model to put as a document .
Here is the mapping
{
"id" : {"type" : "integer" },
"public\_name": {"type" : "string" ,"store" : "no", "index":"no" },
"description": {"type" : "string" ,"store" : "no", "index":"analyzed" },
"website\_url": {"type" : "string" ,"store" : "no", "index":"no" },
"overall\_rating": {"type" : "float" },
"plan\_id": {"type" : "integer" },
"city": {"type" : "string" ,"store" : "yes", "index":"analyzed" },
"zipcode": {"type" : "string" ,"store" : "yes", "index":"analyzed" },
"state": {"type" : "string" ,"store" : "yes", "index":"analyzed" },
"country": {"type" : "string" ,"store" : "yes", "index":"analyzed" },
"location":{"type" : "geo\_point" , "lat\_lon" : true},
"created": {"type" : "date" ,"format" : "YYYY-MM-dd HH:mm:ss" },
```
"profile_type_3" : {
"type" : "object",
"properties" : {
"profile_roles" : {
"properties" : {
"role_id" : {"type" : "integer","store" : "yes",
```
"index":"analyzed"},
"industry\_id" : {"type" : "integer"},
"description" : {"type" : "string","store" : "yes",
"index":"analyzed"}
"skills" : {
"properties" : {
"skill\_id" : {"type" : "integer", "store" : "yes"},
```
"skill_name" : {"type" : "string","store" : "yes",
```
"index":"analyzed"},
"experience\_level\_id" : {"type" : "integer","store"
: "yes"}
}
},
"terms" : {
"properties" : {
"term\_id" : {"type" : "integer","store" : "yes" },
"term\_name" : {"type" : "string","store" : "yes" }
}
}
}
}
}
},
```
"profile_type_2" : {
"type" : "object",
"properties" : {
"profile_roles" : {
"properties" : {
"role_id" : {"type" : "integer","store" : "yes" },
"role_name" : {"type" : "string","store" : "yes" },
"industry_id" : {"type" : "integer"},
"description" : {"type" : "string","store" : "yes",
```
"index":"analyzed"},
"skills" : {
"properties" : {
"skill\_id" : {"type" : "integer","store" : "yes",
"index":"analyzed"},
"skill\_name" : {"type" : "string","store" : "yes",
"index":"analyzed"},
"experience\_level\_id" : {"type" : "integer","store" :
"yes", "index":"analyzed"},
"experience\_level\_name" : {"type" : "string"}
}
}
}
}
}
}
}
Each profile\_type\_\* contains many profile\_roles (as array) and each
profiles\_roles contains many skills (as array) and terms (as array)
So data could look like this
.....
"profile\_type\_3" : {
"profile\_roles" : {
```
{
"role_id" : 1,
"industry_id" : 1,
"description" : "some text"
"skills" : {
{
"skill_id" : 10,
"skill_name" : "PHP",
"experience_level_id" : 1
},
{
"skill_id" : 12,
"skill_name" : "PYTHON",
"experience_level_id" : 3
}
}
"terms" : {
{
"term_id" : 1,
"term_name" : "some text"
},
{
"term_id" : 2,
"term_name" : "some text"
},
}
},
{
"role_id" : 2,
"industry_id" : 13,
"description" : "some text"
"skills" : {
{
"skill_id" : 10,
"skill_name" : "PHP",
"experience_level_id" : 1
},
{
"skill_id" : 12,
"skill_name" : "PYTHON",
"experience_level_id" : 3
}
}
"terms" : {
{
"term_id" : 1,
"term_name" : "some text"
},
{
"term_id" : 2,
"term_name" : "some text"
},
}
},
}
```
}
So a profile\_type\_\* could have many profile\_roles , a rofile\_role could
have many skills and terms
so now if need to get document that have a profile\_roles with role\_id = 1
and a skill\_id = 12 ( within the role\_id = 1) and experience level \>= 2
how can i achieve this ? i read about nested object but here i have nested
nested object profile -\> profiles\_roles -\> skills
For skill I think about creating some kind of tuple field like for example
skills = ["10:1", "12:3"] where the first sohuld be the id and the
second the experience\_id but how to do it for profile\_roles aswell ? by
creating some kind of hash like this : roles = ["1:10:1", "1:12:13"] ?
If someone could help to understand the best way to achieve this it will be
really great, thanks.
--
---
### Author:  [@es\_learner](https://discuss.elastic.co/u/es_learner)
#### Post date: [October 16, 2012, 11:20pm UTC](https://discuss.elastic.co/t/nested-object-with-arrays-best-practice/9378/2 "2012-10-16T23:20:04Z")
I have a similar application.
I would make profile\_roles an array of profile\_role objects and in each profile\_role object, skills and terms are arrays of skill and term object respectively.
Using python,
profile\_roles = [] and each profile\_role is a dict that looks like a {"role\_id": 1, "industry\_id" : 1, "description" : "some text", "skills": [], "terms": []}
skill = {"skill\_id" : 10, "skill\_name" : "PHP", "experience\_level\_id" : 1}
term = {"term\_id" : 1, "term\_name" : "some text"}
Field dereferencing can be done like so: profile\_roles.role\_id, profile\_roles.skills.experience\_level\_id
Hope this helps.
---
### Author:  [@samuel\_merlet](https://discuss.elastic.co/u/samuel_merlet)
#### Post date: [October 17, 2012, 8:16am UTC](https://discuss.elastic.co/t/nested-object-with-arrays-best-practice/9378/3 "2012-10-17T08:16:47Z")
Hi,
Thanks , that help a lot !
But i have false postive by using this way .
if i have 2 roles with different skills and search for a combination ok
role and skill i get a result even if the skill belong to another role
"roles" : {
{
"role\_id" : 2,
"skills" : {
"skill\_id" : 5
}
},
{
"role\_id" : 3,
"skills" : {
"skill\_id" : 4
}
},
}
Now if i filter the query with role id = 2 with skill = 4 ,
( profile\_roles.role\_id = 2 AND profile\_roles.skills.skill\_id = 4 ) i have
a result but it's a false one because role 2 doesn't have a skill 4
Maybe i i need to use nested documents ? Any help bout using this will be
much appreciated.
Thanks
On Tuesday, October 16, 2012 2:31:06 PM UTC+2, samuel merlet wrote:
> Hi,
>
> I have a quite complex model to put as a document .
> Here is the mapping
>
> {
> "id" : {"type" : "integer" },
> "public\_name": {"type" : "string" ,"store" : "no", "index":"no" },
> "description": {"type" : "string" ,"store" : "no", "index":"analyzed"
> },
> "website\_url": {"type" : "string" ,"store" : "no", "index":"no" },
> "overall\_rating": {"type" : "float" },
> "plan\_id": {"type" : "integer" },
> "city": {"type" : "string" ,"store" : "yes", "index":"analyzed" },
> "zipcode": {"type" : "string" ,"store" : "yes", "index":"analyzed" },
> "state": {"type" : "string" ,"store" : "yes", "index":"analyzed" },
> "country": {"type" : "string" ,"store" : "yes", "index":"analyzed" },
> "location":{"type" : "geo\_point" , "lat\_lon" : true},
> "created": {"type" : "date" ,"format" : "YYYY-MM-dd HH:mm:ss" },
>
> ```
> "profile_type_3" : {
> "type" : "object",
> "properties" : {
>
> "profile_roles" : {
> "properties" : {
> "role_id" : {"type" : "integer","store" : "yes",
>
> ```
>
> "index":"analyzed"},
> "industry\_id" : {"type" : "integer"},
> "description" : {"type" : "string","store" : "yes",
> "index":"analyzed"}
> "skills" : {
> "properties" : {
> "skill\_id" : {"type" : "integer", "store" :
> "yes"},
> "skill\_name" : {"type" : "string","store" : "yes",
> "index":"analyzed"},
> "experience\_level\_id" : {"type" :
> "integer","store" : "yes"}
> }
> },
> "terms" : {
> "properties" : {
> "term\_id" : {"type" : "integer","store" : "yes" },
> "term\_name" : {"type" : "string","store" : "yes" }
> }
> }
> }
> }
> }
> },
>
> ```
> "profile_type_2" : {
> "type" : "object",
> "properties" : {
>
> "profile_roles" : {
> "properties" : {
> "role_id" : {"type" : "integer","store" : "yes" },
> "role_name" : {"type" : "string","store" : "yes" },
> "industry_id" : {"type" : "integer"},
> "description" : {"type" : "string","store" : "yes",
>
> ```
>
> "index":"analyzed"},
> "skills" : {
> "properties" : {
> "skill\_id" : {"type" : "integer","store" : "yes",
> "index":"analyzed"},
> "skill\_name" : {"type" : "string","store" : "yes",
> "index":"analyzed"},
> "experience\_level\_id" : {"type" : "integer","store" :
> "yes", "index":"analyzed"},
> "experience\_level\_name" : {"type" : "string"}
> }
> }
> }
> }
> }
> }
>
> }
>
> Each profile\_type\_\* contains many profile\_roles (as array) and each
> profiles\_roles contains many skills (as array) and terms (as array)
>
> So data could look like this
>
> .....
> "profile\_type\_3" : {
>
> "profile\_roles" : {
>
> ```
> {
> "role_id" : 1,
> "industry_id" : 1,
> "description" : "some text"
>
> "skills" : {
> {
> "skill_id" : 10,
> "skill_name" : "PHP",
> "experience_level_id" : 1
> },
> {
> "skill_id" : 12,
> "skill_name" : "PYTHON",
> "experience_level_id" : 3
> }
> }
>
>
> "terms" : {
> {
> "term_id" : 1,
> "term_name" : "some text"
> },
> {
> "term_id" : 2,
> "term_name" : "some text"
> },
> }
> },
>
> {
> "role_id" : 2,
> "industry_id" : 13,
> "description" : "some text"
>
> "skills" : {
> {
> "skill_id" : 10,
> "skill_name" : "PHP",
> "experience_level_id" : 1
> },
> {
> "skill_id" : 12,
> "skill_name" : "PYTHON",
> "experience_level_id" : 3
> }
> }
>
>
> "terms" : {
> {
> "term_id" : 1,
> "term_name" : "some text"
> },
> {
> "term_id" : 2,
> "term_name" : "some text"
> },
> }
> },
>
> }
>
> ```
>
> }
>
> So a profile\_type\_\* could have many profile\_roles , a rofile\_role could
> have many skills and terms
>
> so now if need to get document that have a profile\_roles with role\_id = 1
> and a skill\_id = 12 ( within the role\_id = 1) and experience level \>= 2
> how can i achieve this ? i read about nested object but here i have nested
> nested object profile -\> profiles\_roles -\> skills
>
> For skill I think about creating some kind of tuple field like for example
> skills = ["10:1", "12:3"] where the first sohuld be the id and the
> second the experience\_id but how to do it for profile\_roles aswell ? by
> creating some kind of hash like this : roles = ["1:10:1", "1:12:13"] ?
>
> If someone could help to understand the best way to achieve this it will
> be really great, thanks.
--
---
### Author:  [@samuel\_merlet](https://discuss.elastic.co/u/samuel_merlet)
#### Post date: [October 17, 2012, 8:19am UTC](https://discuss.elastic.co/t/nested-object-with-arrays-best-practice/9378/4 "2012-10-17T08:19:01Z")
Hi,
Thanks , that help a lot !
But i have false postive by using this way .
if i have 2 roles with different skills and search for a combination ok
role and skill i get a result even if the skill belong to another role
"roles" : {
{
"role\_id" : 2,
"skills" : {
"skill\_id" : 5
}
},
{
"role\_id" : 3,
"skills" : {
"skill\_id" : 4
}
},
}
Now if i filter the query with role id = 2 with skill = 4 ,
( profile\_roles.role\_id = 2 AND profile\_roles.skills.skill\_id = 4 ) i have
a result but it's a false one because role 2 doesn't have a skill 4
Maybe i i need to use nested documents ? Any help bout using this will be
much appreciated.
Thanks
--
---
### Author:  [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)
#### Post date: [October 17, 2012, 9:21am UTC](https://discuss.elastic.co/t/nested-object-with-arrays-best-practice/9378/5 "2012-10-17T09:21:22Z")
Hiya
> Thanks , that help a lot !
> But i have false postive by using this way .
> if i have 2 roles with different skills and search for a combination
> ok role and skill i get a result even if the skill belong to another
> role
> Now if i filter the query with role id = 2 with skill = 4 ,
> ( profile\_roles.role\_id = 2 AND profile\_roles.skills.skill\_id = 4 ) i
> have a result but it's a false one because role 2 doesn't have a skill
> 4
> Maybe i i need to use nested documents ? Any help bout using this will
> be much appreciated.
Yes, you need to use nested documents.
To explain why, take this document as an example:
{
stuff: [
{ foo: 1, bar: 1},
{ foo: 2, bar: 2}
]
}
That actually gets flattened to:
{
"stuff.foo": [1, 2],
"stuff.bar": [1, 2]
}
Nested documents keep the nested objects as separate documents
internally. They are not visible as separate documents at the user
level, but that is how they are stored.
Have a look at:
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
For querying, look at:
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
The process is: run a query on the nested docs, then run the rest of the
query on their parent or root docs.
If you need data from the nested docs to also be visible in the parent
or root (ie topmost parent) doc, then you need to include that data in
the levels above, eg using include\_in\_root or include\_in\_parent (see the
nested type docs above), but be aware that the data in the root/parent
will be flattened as per my example above
hth
clint
--
---
### Author:  [@samuel\_merlet](https://discuss.elastic.co/u/samuel_merlet)
#### Post date: [October 17, 2012, 9:35am UTC](https://discuss.elastic.co/t/nested-object-with-arrays-best-practice/9378/6 "2012-10-17T09:35:24Z")
Thanks , it works great.
Can we have nested "nested objects" ?
Also i don't really understand what do you mean by
" If you need data from the nested docs to also be visible in the parent or
root " ?
When i get my results i have already access to the nested docs while
reading the source.
Thanks again
On Wed, Oct 17, 2012 at 11:21 AM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)wrote:
> Hiya
>
> > Thanks , that help a lot !
> > But i have false postive by using this way .
>
> > if i have 2 roles with different skills and search for a combination
> > ok role and skill i get a result even if the skill belong to another
> > role
>
> > Now if i filter the query with role id = 2 with skill = 4 ,
> > ( profile\_roles.role\_id = 2 AND profile\_roles.skills.skill\_id = 4 ) i
> > have a result but it's a false one because role 2 doesn't have a skill
> > 4
>
> > Maybe i i need to use nested documents ? Any help bout using this will
> > be much appreciated.
>
> Yes, you need to use nested documents.
>
> To explain why, take this document as an example:
> {
> stuff: [
> { foo: 1, bar: 1},
> { foo: 2, bar: 2}
> ]
> }
>
> That actually gets flattened to:
>
> {
> "stuff.foo": [1, 2],
> "stuff.bar": [1, 2]
> }
>
> Nested documents keep the nested objects as separate documents
> internally. They are not visible as separate documents at the user
> level, but that is how they are stored.
>
> Have a look at:
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/nested-type.html)
>
> For querying, look at:
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/query-dsl/nested-query.html)
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/query-dsl/nested-filter.html)
>
> The process is: run a query on the nested docs, then run the rest of the
> query on their parent or root docs.
>
> If you need data from the nested docs to also be visible in the parent
> or root (ie topmost parent) doc, then you need to include that data in
> the levels above, eg using include\_in\_root or include\_in\_parent (see the
> nested type docs above), but be aware that the data in the root/parent
> will be flattened as per my example above
>
> hth
>
> clint
>
> --
--
---
### Author:  [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)
#### Post date: [October 17, 2012, 10:13am UTC](https://discuss.elastic.co/t/nested-object-with-arrays-best-practice/9378/7 "2012-10-17T10:13:15Z")
Hiya
> Can we have nested "nested objects" ?
Yes
> Also i don't really understand what do you mean by
> " If you need data from the nested docs to also be visible in the
> parent or root " ?
> When i get my results i have already access to the nested docs while
> reading the source.
That's fine.
What I mean is that: sometimes you want to run queries on the root doc
which involves data stored in the nested docs. Right now I can't think
of a good example 🙂
So, don't worry about it for the moment. Just bear in mind that such an
option exists in case you find yourself unable to do what you need
because the data isn't visible to you at the right level.
clint
> Thanks again
>
> On Wed, Oct 17, 2012 at 11:21 AM, Clinton Gormley
> [clint@traveljury.com](mailto:clint@traveljury.com) wrote:
> Hiya
> \>
> \> Thanks , that help a lot !
> \> But i have false postive by using this way .
>
> ```
> > if i have 2 roles with different skills and search for a
> combination
> > ok role and skill i get a result even if the skill belong to
> another
> > role
>
> > Now if i filter the query with role id = 2 with skill = 4 ,
> > ( profile_roles.role_id = 2 AND
> profile_roles.skills.skill_id = 4 ) i
> > have a result but it's a false one because role 2 doesn't
> have a skill
> > 4
>
> > Maybe i i need to use nested documents ? Any help bout using
> this will
> > be much appreciated.
>
> Yes, you need to use nested documents.
>
> To explain why, take this document as an example:
> {
> stuff: [
> { foo: 1, bar: 1},
> { foo: 2, bar: 2}
> ]
> }
>
> That actually gets flattened to:
>
> {
> "stuff.foo": [1, 2],
> "stuff.bar": [1, 2]
> }
>
>
> Nested documents keep the nested objects as separate documents
> internally. They are not visible as separate documents at the
> user
> level, but that is how they are stored.
>
> Have a look at:
> http://www.elasticsearch.org/guide/reference/mapping/nested-type.html
>
> For querying, look at:
> http://www.elasticsearch.org/guide/reference/query-dsl/nested-query.html
> http://www.elasticsearch.org/guide/reference/query-dsl/nested-filter.html
>
> The process is: run a query on the nested docs, then run the
> rest of the
> query on their parent or root docs.
>
> If you need data from the nested docs to also be visible in
> the parent
> or root (ie topmost parent) doc, then you need to include that
> data in
> the levels above, eg using include_in_root or
> include_in_parent (see the
> nested type docs above), but be aware that the data in the
> root/parent
> will be flattened as per my example above
>
> hth
>
> clint
>
>
>
>
>
>
> --
>
> ```
>
> --
--
---
### Author:  [@samuel\_merlet](https://discuss.elastic.co/u/samuel_merlet)
#### Post date: [October 17, 2012, 3:47pm UTC](https://discuss.elastic.co/t/nested-object-with-arrays-best-practice/9378/8 "2012-10-17T15:47:43Z")
OK , thanks .
From your experiences is it a good practice to do it this way ? Or is there
better solutions like denormalize datas , hash possible combinations or
whatever ..
On Wednesday, October 17, 2012 12:13:34 PM UTC+2, Clinton Gormley wrote:
> Hiya
>
> > Can we have nested "nested objects" ?
>
> Yes
>
> > Also i don't really understand what do you mean by
> > " If you need data from the nested docs to also be visible in the
> > parent or root " ?
> > When i get my results i have already access to the nested docs while
> > reading the source.
>
> That's fine.
>
> What I mean is that: sometimes you want to run queries on the root doc
> which involves data stored in the nested docs. Right now I can't think
> of a good example 🙂
>
> So, don't worry about it for the moment. Just bear in mind that such an
> option exists in case you find yourself unable to do what you need
> because the data isn't visible to you at the right level.
>
> clint
>
> > Thanks again
> >
> > On Wed, Oct 17, 2012 at 11:21 AM, Clinton Gormley
> > \<[cl...@traveljury.com](mailto:cl...@traveljury.com) \\> wrote:
> > Hiya
> > \>
> > \> Thanks , that help a lot !
> > \> But i have false postive by using this way .
> >
> > ```
> > > if i have 2 roles with different skills and search for a
> > combination
> > > ok role and skill i get a result even if the skill belong to
> > another
> > > role
> >
> > > Now if i filter the query with role id = 2 with skill = 4 ,
> > > ( profile_roles.role_id = 2 AND
> > profile_roles.skills.skill_id = 4 ) i
> > > have a result but it's a false one because role 2 doesn't
> > have a skill
> > > 4
> >
> > > Maybe i i need to use nested documents ? Any help bout using
> > this will
> > > be much appreciated.
> >
> > Yes, you need to use nested documents.
> >
> > To explain why, take this document as an example:
> > {
> > stuff: [
> > { foo: 1, bar: 1},
> > { foo: 2, bar: 2}
> > ]
> > }
> >
> > That actually gets flattened to:
> >
> > {
> > "stuff.foo": [1, 2],
> > "stuff.bar": [1, 2]
> > }
> >
> >
> > Nested documents keep the nested objects as separate documents
> > internally. They are not visible as separate documents at the
> > user
> > level, but that is how they are stored.
> >
> > Have a look at:
> >
> > ```
>
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/nested-type.html)
>
> > ```
> > For querying, look at:
> >
> > ```
>
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/query-dsl/nested-query.html)
>
> >
>
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/query-dsl/nested-filter.html)
>
> > ```
> > The process is: run a query on the nested docs, then run the
> > rest of the
> > query on their parent or root docs.
> >
> > If you need data from the nested docs to also be visible in
> > the parent
> > or root (ie topmost parent) doc, then you need to include that
> > data in
> > the levels above, eg using include_in_root or
> > include_in_parent (see the
> > nested type docs above), but be aware that the data in the
> > root/parent
> > will be flattened as per my example above
> >
> > hth
> >
> > clint
> >
> >
> >
> >
> >
> >
> > --
> >
> > ```
> >
> > --
--
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:08am UTC](https://discuss.elastic.co/t/nested-object-with-arrays-best-practice/9378/9 "2017-07-06T03:08:16Z")