# How can I handle duplicate data in Elasticsearch?

**URL:** <https://discuss.elastic.co/t/how-can-i-handle-duplicate-data-in-elasticsearch/10185>\
**Category:** Elasticsearch\
**Created:** [December 27, 2012, 9:40pm UTC](https://discuss.elastic.co/t/how-can-i-handle-duplicate-data-in-elasticsearch/10185 "2012-12-27T21:40:48Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Burak\_Emre\_Kabakci](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/burak_emre_kabakci/32/2212_2.png) [@Burak\_Emre\_Kabakci](https://discuss.elastic.co/u/Burak_Emre_Kabakci)\
**Post date:** [December 27, 2012, 9:40pm UTC](https://discuss.elastic.co/t/how-can-i-handle-duplicate-data-in-elasticsearch/10185/1 "2012-12-27T21:40:48Z")

</div>

I have used parent & child mapping to normalize data but as far as I  
understand there is no way to get any fields from \_parent document.

Here is the mapping of my index:

{  
"mappings": {  
"building": {  
"properties": {  
"name": {  
"type": "string"  
}  
}  
},  
"flat": {  
"\_parent": {  
"type": "building"  
},  
"properties": {  
"name": {  
"type": "string"  
}  
}  
},  
"room": {  
"\_parent": {  
"type": "flat"  
},  
"properties": {  
"name": {  
"type": "string"  
},  
"floor": {  
"type": "long"  
}  
}  
}  
}  
}

Now, I'm trying to find the best way of storing flat\_name and building\_name  
in room type. I won't query these fields but I should be able to get them  
when I query other fields like floor. There will be millions of rooms and I  
don't have much memory so I suspect that these duplicate values may cause  
out of memory. I found \_source field in Elasticsearch documentation and it  
seems good way to store these kind of values. Do you have any efficient  
suggestion for avoiding duplicate values like querying multiple queries or  
hacky way to get fields from \_parent document or denormalized data is the  
only way to handle this kindle of problem?

--

---

<div class="post-metadata">

**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\
**Post date:** [December 28, 2012, 3:05pm UTC](https://discuss.elastic.co/t/how-can-i-handle-duplicate-data-in-elasticsearch/10185/2 "2012-12-28T15:05:18Z")

</div>

Hello,

Assuming you don't have to update rooms and flats too often, you might be  
better off with nested documents.

Back to your original question, you could store the ID of the parent in the  
child when you insert it. Then, you could use the GET or Multi Get[0] API  
to retrieve the parents once you have the children.

Another possibility, which is just as hackish, is to use the IDs of  
children that you search for, and use the has\_child filter[1] to get the  
parent with that ID. You might want to use an IDs query[2] within that.

[0] [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/api/multi-get.html)  
[1]  
[Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/query-dsl/has-child-filter.html)  
[2]  
[Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/query-dsl/has-child-filter.html)

## Best regards, Radu

[http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene

On Thu, Dec 27, 2012 at 11:40 PM, Burak Emre Kabakcı  
[emrekabakci@gmail.com](mailto:emrekabakci@gmail.com)wrote:

> I have used parent & child mapping to normalize data but as far as I  
> understand there is no way to get any fields from \_parent document.
> 
> Here is the mapping of my index:
> 
> {  
> "mappings": {  
> "building": {  
> "properties": {  
> "name": {  
> "type": "string"  
> }  
> }  
> },  
> "flat": {  
> "\_parent": {  
> "type": "building"  
> },  
> "properties": {  
> "name": {  
> "type": "string"  
> }  
> }  
> },  
> "room": {  
> "\_parent": {  
> "type": "flat"  
> },  
> "properties": {  
> "name": {  
> "type": "string"  
> },  
> "floor": {  
> "type": "long"  
> }  
> }  
> }  
> }  
> }
> 
> Now, I'm trying to find the best way of storing flat\_name and  
> building\_name in room type. I won't query these fields but I should be able  
> to get them when I query other fields like floor. There will be millions of  
> rooms and I don't have much memory so I suspect that these duplicate values  
> may cause out of memory. I found \_source field in Elasticsearch  
> documentation and it seems good way to store these kind of values. Do you  
> have any efficient suggestion for avoiding duplicate values like querying  
> multiple queries or hacky way to get fields from \_parent document or  
> denormalized data is the only way to handle this kindle of problem?
> 
> --

--

---

<div class="post-metadata">

**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\
**Post date:** [December 28, 2012, 3:18pm UTC](https://discuss.elastic.co/t/how-can-i-handle-duplicate-data-in-elasticsearch/10185/3 "2012-12-28T15:18:26Z")

</div>

Hi again 🙂

Actually, you don't need to filter to find the parent, since you have the  
parent IDs when you search for children. You just have to add "\_parent" to  
the list of fields you want to be returned when you search. For example:

curl -XPOST localhost:9200/test/room/\_search?pretty=true -d '{  
"fields": ["\_source", "\_parent"],  
"query": {  
"match\_all" :{}  
}  
}'

May return hits like this:  
{  
"\_index" : "test",  
"\_type" : "room",  
"\_id" : "1",  
"\_score" : 1.0, "\_source" : {"name": "room1", "floor": 2},  
"fields" : {  
"\_parent" : "1"  
}  
}

More details here:

> **[Elastic — The Search AI Company](https://www.elastic.co)**
>
> Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ...

So once you have the "child" restults, you can use the Multi Get API to get  
the needed parents. No need to duplicate the parent ID in another field or  
other such hacks.

## Best regards, Radu

[http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene

On Fri, Dec 28, 2012 at 5:05 PM, Radu Gheorghe  
[radu.gheorghe@sematext.com](mailto:radu.gheorghe@sematext.com)wrote:

> Hello,
> 
> Assuming you don't have to update rooms and flats too often, you might be  
> better off with nested documents.
> 
> Back to your original question, you could store the ID of the parent in  
> the child when you insert it. Then, you could use the GET or Multi Get[0]  
> API to retrieve the parents once you have the children.
> 
> Another possibility, which is just as hackish, is to use the IDs of  
> children that you search for, and use the has\_child filter[1] to get the  
> parent with that ID. You might want to use an IDs query[2] within that.
> 
> [0] [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/api/multi-get.html)  
> [1]  
> [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/query-dsl/has-child-filter.html)  
> [2]  
> [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/query-dsl/has-child-filter.html)
> 
> ## Best regards, Radu
> 
> [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene
> 
> On Thu, Dec 27, 2012 at 11:40 PM, Burak Emre Kabakcı \<  
> [emrekabakci@gmail.com](mailto:emrekabakci@gmail.com)\> wrote:
> 
> > I have used parent & child mapping to normalize data but as far as I  
> > understand there is no way to get any fields from \_parent document.
> > 
> > Here is the mapping of my index:
> > 
> > {
> > 
> > "mappings": {  
> > "building": {
> > 
> > ```
> > "properties": {
> > "name": {
> > 
> > "type": "string"
> > }
> > 
> > }
> > },
> > "flat": {
> > 
> > "_parent": {
> > "type": "building"
> > 
> > },
> > "properties": {
> > 
> > "name": {
> > "type": "string"
> > 
> > }
> > }
> > },
> > "room": {
> > 
> > "_parent": {
> > "type": "flat"
> > 
> > },
> > "properties": {
> > 
> > "name": {
> > "type": "string"
> > 
> > },
> > "floor": {
> > 
> > "type": "long"
> > }
> > 
> > }
> > }
> > 
> > ```
> > 
> > }  
> > }
> > 
> > Now, I'm trying to find the best way of storing flat\_name and  
> > building\_name in room type. I won't query these fields but I should be able  
> > to get them when I query other fields like floor. There will be millions of  
> > rooms and I don't have much memory so I suspect that these duplicate values  
> > may cause out of memory. I found \_source field in Elasticsearch  
> > documentation and it seems good way to store these kind of values. Do you  
> > have any efficient suggestion for avoiding duplicate values like querying  
> > multiple queries or hacky way to get fields from \_parent document or  
> > denormalized data is the only way to handle this kindle of problem?
> > 
> > --

--

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:58am UTC](https://discuss.elastic.co/t/how-can-i-handle-duplicate-data-in-elasticsearch/10185/4 "2017-07-06T02:58:18Z")

</div>