# Dealing with large documents (architecture question)

**URL:** <https://discuss.elastic.co/t/dealing-with-large-documents-architecture-question/98404>\
**Category:** Elasticsearch\
**Created:** [August 25, 2017, 6:05pm UTC](https://discuss.elastic.co/t/dealing-with-large-documents-architecture-question/98404 "2017-08-25T18:05:06Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![lauraerinmann](https://avatars.discourse-cdn.com/v4/letter/l/53a042/32.png) [@lauraerinmann](https://discuss.elastic.co/u/lauraerinmann)\
**Post date:** [August 25, 2017, 6:05pm UTC](https://discuss.elastic.co/t/dealing-with-large-documents-architecture-question/98404/1 "2017-08-25T18:05:06Z")

</div>

Hi,  
ElasticSearch's documentation makes it very clear that it's meant to index large documents. I have a document that when in the moderate case is about 15MB and millions of objects. The mapping looks like this:  
{  
"commit": {  
"properties": {  
"added": {  
"type": "nested",  
"properties" : {  
"id":{  
"type": "keyword"  
},  
"elasticId":{  
"type": "keyword"  
}  
}  
},  
"moved": {  
"type": "nested",  
"properties" : {  
"id":{  
"type": "keyword"  
},  
"previousOwner":{  
"properties" : {  
"id":{  
"type": "keyword"  
},  
"elasticId":{  
"type": "keyword"  
}  
}  
},  
"owner":{  
"properties" : {  
"id":{  
"type": "keyword"  
},  
"elasticId":{  
"type": "keyword"  
}  
}  
}  
}  
},  
"updated": {  
"type": "nested",  
"properties" : {  
"id":{  
"type": "keyword"  
},  
"elasticId":{  
"type": "keyword"  
},  
"previousElasticId":{  
"type": "keyword"  
}  
}  
},  
"deleted": {  
"type": "nested",  
"properties" : {  
"id":{  
"type": "keyword"  
},  
"previousElasticId":{  
"type": "keyword"  
}  
}  
},  
"\_creator": {  
"type": "keyword"  
},  
"\_created": {  
"type": "date",  
"format": "yyyy-MM-dd'T'HH:mm:ss.SSSZ"  
}  
}  
}  
}  
Nested objects that are always one level deep.  
Now, my question is given this architecture what is the best way to deal with these large documents (because we have to store them in ES) ie breaking it up, etc? What pros and cons would there be if I was able to get rid of the nested objects?  
There's not a clear way to do this in ES's documentation and I think that's because each case is different. What do you guys think is a practical architecture for this problem?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 26, 2017, 9:58am UTC](https://discuss.elastic.co/t/dealing-with-large-documents-architecture-question/98404/2 "2017-08-26T09:58:43Z")

</div>

> [@lauraerinmann](#):
>
> Elasticsearch’s documentation makes it very clear that it’s meant to index large documents

Do you mean it's **not** meant to? Because it's not really designed for that.

Breaking it up might be the best option, perhaps by the action - ie moved, updated etc.

---

<div class="post-metadata">

**Author:** ![lauraerinmann](https://avatars.discourse-cdn.com/v4/letter/l/53a042/32.png) [@lauraerinmann](https://discuss.elastic.co/u/lauraerinmann)\
**Post date:** [August 28, 2017, 5:19pm UTC](https://discuss.elastic.co/t/dealing-with-large-documents-architecture-question/98404/3 "2017-08-28T17:19:18Z")

</div>

Yes I meant "not", what a typo! So, breaking it up is the best approach for large documents?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 28, 2017, 9:39pm UTC](https://discuss.elastic.co/t/dealing-with-large-documents-architecture-question/98404/4 "2017-08-28T21:39:51Z")

</div>

Yep.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 25, 2017, 9:40pm UTC](https://discuss.elastic.co/t/dealing-with-large-documents-architecture-question/98404/5 "2017-09-25T21:40:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.