# Store Large Object in Index without Mapping

**URL:** <https://discuss.elastic.co/t/store-large-object-in-index-without-mapping/5930>\
**Category:** Elasticsearch\
**Created:** [November 21, 2011, 7:34pm UTC](https://discuss.elastic.co/t/store-large-object-in-index-without-mapping/5930 "2011-11-21T19:34:51Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![davrob](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@davrob](https://discuss.elastic.co/u/davrob)\
**Post date:** [November 21, 2011, 7:34pm UTC](https://discuss.elastic.co/t/store-large-object-in-index-without-mapping/5930/1 "2011-11-21T19:34:51Z")

</div>

Hi,

I would like to store a large hashmap in my index without mapping it  
but having available in the \_source.  
Essentially I want to use ElasticSearch as an object store, where I  
can associate a hashmap with an index id rather than make a call out  
to Mongo DB, and the like, using the id as a reference.

-David.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [November 22, 2011, 8:34am UTC](https://discuss.elastic.co/t/store-large-object-in-index-without-mapping/5930/2 "2011-11-22T08:34:19Z")

</div>

You mean not index that big "hashmap"? You can map the object level  
property of it with enabled set to false, which means it will not even go  
and try and map the object represented by it.

On Mon, Nov 21, 2011 at 9:34 PM, davrob2 [daviroberts@gmail.com](mailto:daviroberts@gmail.com) wrote:

> Hi,
> 
> I would like to store a large hashmap in my index without mapping it  
> but having available in the \_source.  
> Essentially I want to use Elasticsearch as an object store, where I  
> can associate a hashmap with an index id rather than make a call out  
> to Mongo DB, and the like, using the id as a reference.
> 
> -David.

---

<div class="post-metadata">

**Author:** ![davrob](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@davrob](https://discuss.elastic.co/u/davrob)\
**Post date:** [November 23, 2011, 11:38am UTC](https://discuss.elastic.co/t/store-large-object-in-index-without-mapping/5930/3 "2011-11-23T11:38:10Z")

</div>

Thanks Shay, that works great.

On Nov 22, 8:34 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:

> You mean not index that big "hashmap"? You can map the object level  
> property of it with enabled set to false, which means it will not even go  
> and try and map the object represented by it.
> 
> On Mon, Nov 21, 2011 at 9:34 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote:
> 
> > Hi,
> 
> > I would like to store a large hashmap in my index without mapping it  
> > but having available in the \_source.  
> > Essentially I want to use Elasticsearch as an object store, where I  
> > can associate a hashmap with an index id rather than make a call out  
> > to Mongo DB, and the like, using the id as a reference.
> 
> > -David.

---

<div class="post-metadata">

**Author:** ![davrob](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@davrob](https://discuss.elastic.co/u/davrob)\
**Post date:** [November 24, 2011, 5:08pm UTC](https://discuss.elastic.co/t/store-large-object-in-index-without-mapping/5930/4 "2011-11-24T17:08:16Z")

</div>

Having implemented the solution using enabled=false on a normal object  
type - I find that my query time has increased massively, probably  
because the queries I execute almost always access the "source"  
object, which now has to deserialize additional very big objects.

## From the Nested Object Documentation

"Those internal nested documents are automatically masked away when  
doing operations against the index (like searching with a match\_all  
query), and they bubble out when using the nested query."

So, my idea is, to offer users the functionality to include this  
object (using a checkbox option) by pushing the large HashMap into a  
nested property where, as I understand it, nested objects are only de-  
serialized if a query is made against them.

My Ideas is to define the mapping, as follows:

{  
"MainType" : {  
"properties" : {  
"nestedObject" : {  
"type" : "nested",  
"properties" : {  
"constantField" : {"type" : "string", "index" : "not\_analyzed"},  
"weekViewMap" : {"type" : "object", "enabled" : false}  
}  
}  
}  
}  
}

So, I would "turn on" the weekView serialization, by making a Nested  
TermFilter search, where nestedObject.contstantField="ALL-OBJECTS-HAVE-  
THIS".

## Questions

1. Will this work?
2. Will a query against another nested object also cause this nested  
object to be included in the source, or do the different nested  
objects act indepently?

- David

On Nov 22, 8:34 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:

> You mean not index that big "hashmap"? You can map the object level  
> property of it with enabled set to false, which means it will not even go  
> and try and map the object represented by it.
> 
> On Mon, Nov 21, 2011 at 9:34 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote:
> 
> > Hi,
> 
> > I would like to store a large hashmap in my index without mapping it  
> > but having available in the \_source.  
> > Essentially I want to use Elasticsearch as an object store, where I  
> > can associate a hashmap with an index id rather than make a call out  
> > to Mongo DB, and the like, using the id as a reference.
> 
> > -David.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [November 24, 2011, 6:26pm UTC](https://discuss.elastic.co/t/store-large-object-in-index-without-mapping/5930/5 "2011-11-24T18:26:40Z")

</div>

You mean the object is new that you now add to your document? I suggest you  
first understand why its slower, for example, by asking for empty fields,  
or even \_source.

On Thu, Nov 24, 2011 at 7:08 PM, davrob2 [daviroberts@gmail.com](mailto:daviroberts@gmail.com) wrote:

> Having implemented the solution using enabled=false on a normal object  
> type - I find that my query time has increased massively, probably  
> because the queries I execute almost always access the "source"  
> object, which now has to deserialize additional very big objects.
> 
> ## From the Nested Object Documentation
> 
> "Those internal nested documents are automatically masked away when  
> doing operations against the index (like searching with a match\_all  
> query), and they bubble out when using the nested query."
> 
> So, my idea is, to offer users the functionality to include this  
> object (using a checkbox option) by pushing the large HashMap into a  
> nested property where, as I understand it, nested objects are only de-  
> serialized if a query is made against them.
> 
> My Ideas is to define the mapping, as follows:
> 
> {  
> "MainType" : {  
> "properties" : {  
> "nestedObject" : {  
> "type" : "nested",  
> "properties" : {  
> "constantField" : {"type" :  
> "string", "index" : "not\_analyzed"},  
> "weekViewMap" : {"type" : "object",  
> "enabled" : false}  
> }  
> }  
> }  
> }  
> }
> 
> So, I would "turn on" the weekView serialization, by making a Nested  
> TermFilter search, where nestedObject.contstantField="ALL-OBJECTS-HAVE-  
> THIS".
> 
> ## Questions
> 
> 1. Will this work?
> 2. Will a query against another nested object also cause this nested  
> object to be included in the source, or do the different nested  
> objects act indepently?
> 
> - David
> 
> On Nov 22, 8:34 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> 
> > You mean not index that big "hashmap"? You can map the object level  
> > property of it with enabled set to false, which means it will not even go  
> > and try and map the object represented by it.
> > 
> > On Mon, Nov 21, 2011 at 9:34 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote:
> > 
> > > Hi,
> > 
> > > I would like to store a large hashmap in my index without mapping it  
> > > but having available in the \_source.  
> > > Essentially I want to use Elasticsearch as an object store, where I  
> > > can associate a hashmap with an index id rather than make a call out  
> > > to Mongo DB, and the like, using the id as a reference.
> > 
> > > -David.

---

<div class="post-metadata">

**Author:** ![davrob](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@davrob](https://discuss.elastic.co/u/davrob)\
**Post date:** [November 25, 2011, 11:08am UTC](https://discuss.elastic.co/t/store-large-object-in-index-without-mapping/5930/6 "2011-11-25T11:08:42Z")

</div>

Hi Shay,

So these are the stats for this operation:

```
Timer timer = Timer.startTimer();
SearchResponse searchResponse = searchReq.execute().actionGet();
timer.endTimer("QUERY TIME --->>>");

```

# Stats

1. All fields but 'large' field (obtained by running against an old  
index without this in):

2. One field - obtained from the index with the 'large' field in, but  
just asking for one field using "searchReq.addFields("person\_name");"

3. All fields including 'large' field:

I think that proves that it is the serialization that is causing the  
problem, rather than the search.

In that case, do you think the above approach will be a good way of  
stoping that object being serilized by default?

Or, a good alternative would be to have a method on  
SearchRequestBuilder like:

searchReq.removeFields("weekViewMap");

```
         or

```

searchReq.doNotSerializeFields("weekViewMap");

For the default (fields = null), which pulls back the source.

-David

On Nov 24, 6:26 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:

> You mean the object is new that you now add to your document? I suggest you  
> first understand why its slower, for example, by asking for empty fields,  
> or even \_source.
> 
> On Thu, Nov 24, 2011 at 7:08 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote:
> 
> > Having implemented the solution using enabled=false on a normal object  
> > type - I find that my query time has increased massively, probably  
> > because the queries I execute almost always access the "source"  
> > object, which now has to deserialize additional very big objects.
> 
> > ## From the Nested Object Documentation
> > 
> > "Those internal nested documents are automatically masked away when  
> > doing operations against the index (like searching with a match\_all  
> > query), and they bubble out when using the nested query."
> 
> > So, my idea is, to offer users the functionality to include this  
> > object (using a checkbox option) by pushing the large HashMap into a  
> > nested property where, as I understand it, nested objects are only de-  
> > serialized if a query is made against them.
> 
> > My Ideas is to define the mapping, as follows:
> 
> > {  
> > "MainType" : {  
> > "properties" : {  
> > "nestedObject" : {  
> > "type" : "nested",  
> > "properties" : {  
> > "constantField" : {"type" :  
> > "string", "index" : "not\_analyzed"},  
> > "weekViewMap" : {"type" : "object",  
> > "enabled" : false}  
> > }  
> > }  
> > }  
> > }  
> > }
> 
> > So, I would "turn on" the weekView serialization, by making a Nested  
> > TermFilter search, where nestedObject.contstantField="ALL-OBJECTS-HAVE-  
> > THIS".
> 
> > ## Questions
> > 
> > 1. Will this work?
> > 2. Will a query against another nested object also cause this nested  
> > object to be included in the source, or do the different nested  
> > objects act indepently?
> 
> > - David
> 
> > On Nov 22, 8:34 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > 
> > > You mean not index that big "hashmap"? You can map the object level  
> > > property of it with enabled set to false, which means it will not even go  
> > > and try and map the object represented by it.
> 
> > > On Mon, Nov 21, 2011 at 9:34 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote:
> > > 
> > > > Hi,
> 
> > > > I would like to store a large hashmap in my index without mapping it  
> > > > but having available in the \_source.  
> > > > Essentially I want to use Elasticsearch as an object store, where I  
> > > > can associate a hashmap with an index id rather than make a call out  
> > > > to Mongo DB, and the like, using the id as a reference.
> 
> > > > -David.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [November 27, 2011, 3:20pm UTC](https://discuss.elastic.co/t/store-large-object-in-index-without-mapping/5930/7 "2011-11-27T15:20:03Z")

</div>

Do you rely on \_source or do you specifically store fields? If so, then I  
don't really understand this big difference, since when you ask for  
specific fields, it still gets loaded and parsed (with the "large" field),  
and if you don't, it just a matter of passing the \_source as is through the  
wire.

On Fri, Nov 25, 2011 at 1:08 PM, davrob2 [daviroberts@gmail.com](mailto:daviroberts@gmail.com) wrote:

> Hi Shay,
> 
> So these are the stats for this operation:
> 
> Timer timer = Timer.startTimer();  
> SearchResponse searchResponse = searchReq.execute().actionGet();  
> timer.endTimer("QUERY TIME ---\>\>\>");
> 
> # Stats
> 
> 1. All fields but 'large' field (obtained by running against an old  
> index without this in):
> 
> 2. One field - obtained from the index with the 'large' field in, but  
> just asking for one field using "searchReq.addFields("person\_name");"
> 
> 3. All fields including 'large' field:
> 
> I think that proves that it is the serialization that is causing the  
> problem, rather than the search.
> 
> In that case, do you think the above approach will be a good way of  
> stoping that object being serilized by default?
> 
> Or, a good alternative would be to have a method on  
> SearchRequestBuilder like:
> 
> searchReq.removeFields("weekViewMap");
> 
> ```
> or
> 
> ```
> 
> searchReq.doNotSerializeFields("weekViewMap");
> 
> For the default (fields = null), which pulls back the source.
> 
> -David
> 
> On Nov 24, 6:26 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> 
> > You mean the object is new that you now add to your document? I suggest  
> > you  
> > first understand why its slower, for example, by asking for empty fields,  
> > or even \_source.
> > 
> > On Thu, Nov 24, 2011 at 7:08 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote:
> > 
> > > Having implemented the solution using enabled=false on a normal object  
> > > type - I find that my query time has increased massively, probably  
> > > because the queries I execute almost always access the "source"  
> > > object, which now has to deserialize additional very big objects.
> > 
> > > ## From the Nested Object Documentation
> > > 
> > > "Those internal nested documents are automatically masked away when  
> > > doing operations against the index (like searching with a match\_all  
> > > query), and they bubble out when using the nested query."
> > 
> > > So, my idea is, to offer users the functionality to include this  
> > > object (using a checkbox option) by pushing the large HashMap into a  
> > > nested property where, as I understand it, nested objects are only de-  
> > > serialized if a query is made against them.
> > 
> > > My Ideas is to define the mapping, as follows:
> > 
> > > {  
> > > "MainType" : {  
> > > "properties" : {  
> > > "nestedObject" : {  
> > > "type" : "nested",  
> > > "properties" : {  
> > > "constantField" : {"type" :  
> > > "string", "index" : "not\_analyzed"},  
> > > "weekViewMap" : {"type" :  
> > > "object",  
> > > "enabled" : false}  
> > > }  
> > > }  
> > > }  
> > > }  
> > > }
> > 
> > > So, I would "turn on" the weekView serialization, by making a Nested  
> > > TermFilter search, where nestedObject.contstantField="ALL-OBJECTS-HAVE-  
> > > THIS".
> > 
> > > ## Questions
> > > 
> > > 1. Will this work?
> > > 2. Will a query against another nested object also cause this nested  
> > > object to be included in the source, or do the different nested  
> > > objects act indepently?
> > 
> > > - David
> > 
> > > On Nov 22, 8:34 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > > 
> > > > You mean not index that big "hashmap"? You can map the object level  
> > > > property of it with enabled set to false, which means it will not  
> > > > even go  
> > > > and try and map the object represented by it.
> > 
> > > > On Mon, Nov 21, 2011 at 9:34 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com)  
> > > > wrote:
> > > > 
> > > > > Hi,
> > 
> > > > > I would like to store a large hashmap in my index without mapping  
> > > > > it  
> > > > > but having available in the \_source.  
> > > > > Essentially I want to use Elasticsearch as an object store, where I  
> > > > > can associate a hashmap with an index id rather than make a call  
> > > > > out  
> > > > > to Mongo DB, and the like, using the id as a reference.
> > 
> > > > > -David.

---

<div class="post-metadata">

**Author:** ![davrob](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@davrob](https://discuss.elastic.co/u/davrob)\
**Post date:** [November 28, 2011, 10:06am UTC](https://discuss.elastic.co/t/store-large-object-in-index-without-mapping/5930/8 "2011-11-28T10:06:08Z")

</div>

Hi Shay,

Yes, I do rely on \_source usually, but when I specify the field to  
search using "searchReq.addFields("person\_name");" then the source  
returned is null and I just get the individual field as below:

```
for (SearchHit hit : queryResponse.getHits()){
 Map<String, Object> source = hit.getSource();
     if (source == null){
    source = new HashMap<String, Object>();
    for (String responseField :

```

searchBean.getSearch().getResponseFields()){  
SearchHitField shf = hit.field(responseField);  
if (shf != null){  
source.put(responseField, shf.getValue());  
}  
}  
}

- David

On Nov 27, 3:20 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:

> Do you rely on \_source or do you specifically store fields? If so, then I  
> don't really understand this big difference, since when you ask for  
> specific fields, it still gets loaded and parsed (with the "large" field),  
> and if you don't, it just a matter of passing the \_source as is through the  
> wire.
> 
> On Fri, Nov 25, 2011 at 1:08 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote:
> 
> > Hi Shay,
> 
> > So these are the stats for this operation:
> 
> > Timer timer = Timer.startTimer();  
> > SearchResponse searchResponse = searchReq.execute().actionGet();  
> > timer.endTimer("QUERY TIME ---\>\>\>");
> 
> > # Stats
> > 
> > 1. All fields but 'large' field (obtained by running against an old  
> > index without this in):
> 
> > INFO: QUERY TIME ---\>\>\>366ms
> 
> > 1. One field - obtained from the index with the 'large' field in, but  
> > just asking for one field using "searchReq.addFields("person\_name");"
> 
> > INFO: QUERY TIME ---\>\>\>344ms
> 
> > 1. All fields including 'large' field:
> 
> > INFO: QUERY TIME ---\>\>\>2922ms
> 
> > I think that proves that it is the serialization that is causing the  
> > problem, rather than the search.
> 
> > In that case, do you think the above approach will be a good way of  
> > stoping that object being serilized by default?
> 
> > Or, a good alternative would be to have a method on  
> > SearchRequestBuilder like:
> 
> > searchReq.removeFields("weekViewMap");
> 
> > ```
> > or
> > 
> > ```
> 
> > searchReq.doNotSerializeFields("weekViewMap");
> 
> > For the default (fields = null), which pulls back the source.
> 
> > -David
> 
> > On Nov 24, 6:26 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > 
> > > You mean the object is new that you now add to your document? I suggest  
> > > you  
> > > first understand why its slower, for example, by asking for empty fields,  
> > > or even \_source.
> 
> > > On Thu, Nov 24, 2011 at 7:08 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote:
> > > 
> > > > Having implemented the solution using enabled=false on a normal object  
> > > > type - I find that my query time has increased massively, probably  
> > > > because the queries I execute almost always access the "source"  
> > > > object, which now has to deserialize additional very big objects.
> 
> > > > ## From the Nested Object Documentation
> > > > 
> > > > "Those internal nested documents are automatically masked away when  
> > > > doing operations against the index (like searching with a match\_all  
> > > > query), and they bubble out when using the nested query."
> 
> > > > So, my idea is, to offer users the functionality to include this  
> > > > object (using a checkbox option) by pushing the large HashMap into a  
> > > > nested property where, as I understand it, nested objects are only de-  
> > > > serialized if a query is made against them.
> 
> > > > My Ideas is to define the mapping, as follows:
> 
> > > > {  
> > > > "MainType" : {  
> > > > "properties" : {  
> > > > "nestedObject" : {  
> > > > "type" : "nested",  
> > > > "properties" : {  
> > > > "constantField" : {"type" :  
> > > > "string", "index" : "not\_analyzed"},  
> > > > "weekViewMap" : {"type" :  
> > > > "object",  
> > > > "enabled" : false}  
> > > > }  
> > > > }  
> > > > }  
> > > > }  
> > > > }
> 
> > > > So, I would "turn on" the weekView serialization, by making a Nested  
> > > > TermFilter search, where nestedObject.contstantField="ALL-OBJECTS-HAVE-  
> > > > THIS".
> 
> > > > ## Questions
> > > > 
> > > > 1. Will this work?
> > > > 2. Will a query against another nested object also cause this nested  
> > > > object to be included in the source, or do the different nested  
> > > > objects act indepently?
> 
> > > > - David
> 
> > > > On Nov 22, 8:34 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > > > 
> > > > > You mean not index that big "hashmap"? You can map the object level  
> > > > > property of it with enabled set to false, which means it will not  
> > > > > even go  
> > > > > and try and map the object represented by it.
> 
> > > > > On Mon, Nov 21, 2011 at 9:34 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com)  
> > > > > wrote:
> > > > > 
> > > > > > Hi,
> 
> > > > > > I would like to store a large hashmap in my index without mapping  
> > > > > > it  
> > > > > > but having available in the \_source.  
> > > > > > Essentially I want to use Elasticsearch as an object store, where I  
> > > > > > can associate a hashmap with an index id rather than make a call  
> > > > > > out  
> > > > > > to Mongo DB, and the like, using the id as a reference.
> 
> > > > > > -David.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [November 28, 2011, 11:23am UTC](https://discuss.elastic.co/t/store-large-object-in-index-without-mapping/5930/9 "2011-11-28T11:23:46Z")

</div>

Right, but it still gets loaded and extracted, the \_source, when you ask  
for a field. And, just transferring it seems strange that it will cause  
that big of a time difference. How big is the document that you index (with  
that \_map)?

On Mon, Nov 28, 2011 at 12:06 PM, davrob2 [daviroberts@gmail.com](mailto:daviroberts@gmail.com) wrote:

> Hi Shay,
> 
> Yes, I do rely on \_source usually, but when I specify the field to  
> search using "searchReq.addFields("person\_name");" then the source  
> returned is null and I just get the individual field as below:
> 
> for (SearchHit hit : queryResponse.getHits()){  
> Map\<String, Object\> source = hit.getSource();  
> if (source == null){  
> source = new HashMap\<String, Object\>();  
> for (String responseField :  
> searchBean.getSearch().getResponseFields()){  
> SearchHitField shf = hit.field(responseField);  
> if (shf != null){  
> source.put(responseField, shf.getValue());  
> }  
> }  
> }
> 
> - David
> 
> On Nov 27, 3:20 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> 
> > Do you rely on \_source or do you specifically store fields? If so, then I  
> > don't really understand this big difference, since when you ask for  
> > specific fields, it still gets loaded and parsed (with the "large"  
> > field),  
> > and if you don't, it just a matter of passing the \_source as is through  
> > the  
> > wire.
> > 
> > On Fri, Nov 25, 2011 at 1:08 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote:
> > 
> > > Hi Shay,
> > 
> > > So these are the stats for this operation:
> > 
> > > Timer timer = Timer.startTimer();  
> > > SearchResponse searchResponse = searchReq.execute().actionGet();  
> > > timer.endTimer("QUERY TIME ---\>\>\>");
> > 
> > > # Stats
> > > 
> > > 1. All fields but 'large' field (obtained by running against an old  
> > > index without this in):
> > 
> > > INFO: QUERY TIME ---\>\>\>366ms
> > 
> > > 1. One field - obtained from the index with the 'large' field in, but  
> > > just asking for one field using "searchReq.addFields("person\_name");"
> > 
> > > INFO: QUERY TIME ---\>\>\>344ms
> > 
> > > 1. All fields including 'large' field:
> > 
> > > INFO: QUERY TIME ---\>\>\>2922ms
> > 
> > > I think that proves that it is the serialization that is causing the  
> > > problem, rather than the search.
> > 
> > > In that case, do you think the above approach will be a good way of  
> > > stoping that object being serilized by default?
> > 
> > > Or, a good alternative would be to have a method on  
> > > SearchRequestBuilder like:
> > 
> > > searchReq.removeFields("weekViewMap");
> > 
> > > ```
> > > or
> > > 
> > > ```
> > 
> > > searchReq.doNotSerializeFields("weekViewMap");
> > 
> > > For the default (fields = null), which pulls back the source.
> > 
> > > -David
> > 
> > > On Nov 24, 6:26 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > > 
> > > > You mean the object is new that you now add to your document? I  
> > > > suggest  
> > > > you  
> > > > first understand why its slower, for example, by asking for empty  
> > > > fields,  
> > > > or even \_source.
> > 
> > > > On Thu, Nov 24, 2011 at 7:08 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com)  
> > > > wrote:
> > > > 
> > > > > Having implemented the solution using enabled=false on a normal  
> > > > > object  
> > > > > type - I find that my query time has increased massively, probably  
> > > > > because the queries I execute almost always access the "source"  
> > > > > object, which now has to deserialize additional very big objects.
> > 
> > > > > ## From the Nested Object Documentation
> > > > > 
> > > > > "Those internal nested documents are automatically masked away when  
> > > > > doing operations against the index (like searching with a match\_all  
> > > > > query), and they bubble out when using the nested query."
> > 
> > > > > So, my idea is, to offer users the functionality to include this  
> > > > > object (using a checkbox option) by pushing the large HashMap into  
> > > > > a  
> > > > > nested property where, as I understand it, nested objects are only  
> > > > > de-  
> > > > > serialized if a query is made against them.
> > 
> > > > > My Ideas is to define the mapping, as follows:
> > 
> > > > > {  
> > > > > "MainType" : {  
> > > > > "properties" : {  
> > > > > "nestedObject" : {  
> > > > > "type" : "nested",  
> > > > > "properties" : {  
> > > > > "constantField" : {"type" :  
> > > > > "string", "index" : "not\_analyzed"},  
> > > > > "weekViewMap" : {"type" :  
> > > > > "object",  
> > > > > "enabled" : false}  
> > > > > }  
> > > > > }  
> > > > > }  
> > > > > }  
> > > > > }
> > 
> > > > > So, I would "turn on" the weekView serialization, by making a  
> > > > > Nested  
> > > > > TermFilter search, where  
> > > > > nestedObject.contstantField="ALL-OBJECTS-HAVE-  
> > > > > THIS".
> > 
> > > > > ## Questions
> > > > > 
> > > > > 1. Will this work?
> > > > > 2. Will a query against another nested object also cause this  
> > > > > nested  
> > > > > object to be included in the source, or do the different nested  
> > > > > objects act indepently?
> > 
> > > > > - David
> > 
> > > > > On Nov 22, 8:34 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > > > > 
> > > > > > You mean not index that big "hashmap"? You can map the object  
> > > > > > level  
> > > > > > property of it with enabled set to false, which means it will not  
> > > > > > even go  
> > > > > > and try and map the object represented by it.
> > 
> > > > > > On Mon, Nov 21, 2011 at 9:34 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com)  
> > > > > > wrote:
> > > > > > 
> > > > > > > Hi,
> > 
> > > > > > > I would like to store a large hashmap in my index without  
> > > > > > > mapping  
> > > > > > > it  
> > > > > > > but having available in the \_source.  
> > > > > > > Essentially I want to use Elasticsearch as an object store,  
> > > > > > > where I  
> > > > > > > can associate a hashmap with an index id rather than make a  
> > > > > > > call  
> > > > > > > out  
> > > > > > > to Mongo DB, and the like, using the id as a reference.
> > 
> > > > > > > -David.

---

<div class="post-metadata">

**Author:** ![davrob](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@davrob](https://discuss.elastic.co/u/davrob)\
**Post date:** [November 28, 2011, 4:10pm UTC](https://discuss.elastic.co/t/store-large-object-in-index-without-mapping/5930/10 "2011-11-28T16:10:35Z")

</div>

Hi Shay,

Well, yes, I think your instinct is right, the difference in the files  
is only very small, here's an example:

> <https://gist.github.com/dav-rob/1400795>

The Average size of Contact document is 1327 characters, while the  
average size of the contacts with the added hashmap is 1699  
characters.

So, I'm not sure what is causing the issue, I do have a map within the  
map, but I'd have thought the JSON serialization API would have no  
real problem with that.

The work around I have in mind at the moment is to have a "fat" type  
"fatcontact" and a skinny one "contact" that are identical apart from  
the embedded HashMap. If users want to search on the embedded map I  
switch types to search on "fatcontact" type - but this is a pretty  
ugly work-around, because I index everything twice.

-David.

On Nov 28, 11:23 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:

> Right, but it still gets loaded and extracted, the \_source, when you ask  
> for a field. And, just transferring it seems strange that it will cause  
> that big of a time difference. How big is the document that you index (with  
> that \_map)?
> 
> On Mon, Nov 28, 2011 at 12:06 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote:
> 
> > Hi Shay,
> 
> > Yes, I do rely on \_source usually, but when I specify the field to  
> > search using "searchReq.addFields("person\_name");" then the source  
> > returned is null and I just get the individual field as below:
> 
> > for (SearchHit hit : queryResponse.getHits()){  
> > Map\<String, Object\> source = hit.getSource();  
> > if (source == null){  
> > source = new HashMap\<String, Object\>();  
> > for (String responseField :  
> > searchBean.getSearch().getResponseFields()){  
> > SearchHitField shf = hit.field(responseField);  
> > if (shf != null){  
> > source.put(responseField, shf.getValue());  
> > }  
> > }  
> > }
> 
> > - David
> 
> > On Nov 27, 3:20 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > 
> > > Do you rely on \_source or do you specifically store fields? If so, then I  
> > > don't really understand this big difference, since when you ask for  
> > > specific fields, it still gets loaded and parsed (with the "large"  
> > > field),  
> > > and if you don't, it just a matter of passing the \_source as is through  
> > > the  
> > > wire.
> 
> > > On Fri, Nov 25, 2011 at 1:08 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote:
> > > 
> > > > Hi Shay,
> 
> > > > So these are the stats for this operation:
> 
> > > > Timer timer = Timer.startTimer();  
> > > > SearchResponse searchResponse = searchReq.execute().actionGet();  
> > > > timer.endTimer("QUERY TIME ---\>\>\>");
> 
> > > > # Stats
> > > > 
> > > > 1. All fields but 'large' field (obtained by running against an old  
> > > > index without this in):
> 
> > > > INFO: QUERY TIME ---\>\>\>366ms
> 
> > > > 1. One field - obtained from the index with the 'large' field in, but  
> > > > just asking for one field using "searchReq.addFields("person\_name");"
> 
> > > > INFO: QUERY TIME ---\>\>\>344ms
> 
> > > > 1. All fields including 'large' field:
> 
> > > > INFO: QUERY TIME ---\>\>\>2922ms
> 
> > > > I think that proves that it is the serialization that is causing the  
> > > > problem, rather than the search.
> 
> > > > In that case, do you think the above approach will be a good way of  
> > > > stoping that object being serilized by default?
> 
> > > > Or, a good alternative would be to have a method on  
> > > > SearchRequestBuilder like:
> 
> > > > searchReq.removeFields("weekViewMap");
> 
> > > > ```
> > > > or
> > > > 
> > > > ```
> 
> > > > searchReq.doNotSerializeFields("weekViewMap");
> 
> > > > For the default (fields = null), which pulls back the source.
> 
> > > > -David
> 
> > > > On Nov 24, 6:26 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > > > 
> > > > > You mean the object is new that you now add to your document? I  
> > > > > suggest  
> > > > > you  
> > > > > first understand why its slower, for example, by asking for empty  
> > > > > fields,  
> > > > > or even \_source.
> 
> > > > > On Thu, Nov 24, 2011 at 7:08 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com)  
> > > > > wrote:
> > > > > 
> > > > > > Having implemented the solution using enabled=false on a normal  
> > > > > > object  
> > > > > > type - I find that my query time has increased massively, probably  
> > > > > > because the queries I execute almost always access the "source"  
> > > > > > object, which now has to deserialize additional very big objects.
> 
> > > > > > ## From the Nested Object Documentation
> > > > > > 
> > > > > > "Those internal nested documents are automatically masked away when  
> > > > > > doing operations against the index (like searching with a match\_all  
> > > > > > query), and they bubble out when using the nested query."
> 
> > > > > > So, my idea is, to offer users the functionality to include this  
> > > > > > object (using a checkbox option) by pushing the large HashMap into  
> > > > > > a  
> > > > > > nested property where, as I understand it, nested objects are only  
> > > > > > de-  
> > > > > > serialized if a query is made against them.
> 
> > > > > > My Ideas is to define the mapping, as follows:
> 
> > > > > > {  
> > > > > > "MainType" : {  
> > > > > > "properties" : {  
> > > > > > "nestedObject" : {  
> > > > > > "type" : "nested",  
> > > > > > "properties" : {  
> > > > > > "constantField" : {"type" :  
> > > > > > "string", "index" : "not\_analyzed"},  
> > > > > > "weekViewMap" : {"type" :  
> > > > > > "object",  
> > > > > > "enabled" : false}  
> > > > > > }  
> > > > > > }  
> > > > > > }  
> > > > > > }  
> > > > > > }
> 
> > > > > > So, I would "turn on" the weekView serialization, by making a  
> > > > > > Nested  
> > > > > > TermFilter search, where  
> > > > > > nestedObject.contstantField="ALL-OBJECTS-HAVE-  
> > > > > > THIS".
> 
> > > > > > ## Questions
> > > > > > 
> > > > > > 1. Will this work?
> > > > > > 2. Will a query against another nested object also cause this  
> > > > > > nested  
> > > > > > object to be included in the source, or do the different nested  
> > > > > > objects act indepently?
> 
> > > > > > - David
> 
> > > > > > On Nov 22, 8:34 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > > > > > 
> > > > > > > You mean not index that big "hashmap"? You can map the object  
> > > > > > > level  
> > > > > > > property of it with enabled set to false, which means it will not  
> > > > > > > even go  
> > > > > > > and try and map the object represented by it.
> 
> > > > > > > On Mon, Nov 21, 2011 at 9:34 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com)  
> > > > > > > wrote:
> > > > > > > 
> > > > > > > > Hi,
> 
> > > > > > > > I would like to store a large hashmap in my index without  
> > > > > > > > mapping  
> > > > > > > > it  
> > > > > > > > but having available in the \_source.  
> > > > > > > > Essentially I want to use Elasticsearch as an object store,  
> > > > > > > > where I  
> > > > > > > > can associate a hashmap with an index id rather than make a  
> > > > > > > > call  
> > > > > > > > out  
> > > > > > > > to Mongo DB, and the like, using the id as a reference.
> 
> > > > > > > > -David.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [November 28, 2011, 4:27pm UTC](https://discuss.elastic.co/t/store-large-object-in-index-without-mapping/5930/11 "2011-11-28T16:27:12Z")

</div>

If thats the case, then its really strange. Can you post a recreation that  
shows this (can be Java code, with a simple "main class" repro)?

On Mon, Nov 28, 2011 at 6:10 PM, davrob2 [daviroberts@gmail.com](mailto:daviroberts@gmail.com) wrote:

> Hi Shay,
> 
> Well, yes, I think your instinct is right, the difference in the files  
> is only very small, here's an example:
> 
> [Week View Map · GitHub](https://gist.github.com/1400795)
> 
> The Average size of Contact document is 1327 characters, while the  
> average size of the contacts with the added hashmap is 1699  
> characters.
> 
> So, I'm not sure what is causing the issue, I do have a map within the  
> map, but I'd have thought the JSON serialization API would have no  
> real problem with that.
> 
> The work around I have in mind at the moment is to have a "fat" type  
> "fatcontact" and a skinny one "contact" that are identical apart from  
> the embedded HashMap. If users want to search on the embedded map I  
> switch types to search on "fatcontact" type - but this is a pretty  
> ugly work-around, because I index everything twice.
> 
> -David.
> 
> On Nov 28, 11:23 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> 
> > Right, but it still gets loaded and extracted, the \_source, when you ask  
> > for a field. And, just transferring it seems strange that it will cause  
> > that big of a time difference. How big is the document that you index  
> > (with  
> > that \_map)?
> > 
> > On Mon, Nov 28, 2011 at 12:06 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote:
> > 
> > > Hi Shay,
> > 
> > > Yes, I do rely on \_source usually, but when I specify the field to  
> > > search using "searchReq.addFields("person\_name");" then the source  
> > > returned is null and I just get the individual field as below:
> > 
> > > for (SearchHit hit : queryResponse.getHits()){  
> > > Map\<String, Object\> source = hit.getSource();  
> > > if (source == null){  
> > > source = new HashMap\<String, Object\>();  
> > > for (String responseField :  
> > > searchBean.getSearch().getResponseFields()){  
> > > SearchHitField shf = hit.field(responseField);  
> > > if (shf != null){  
> > > source.put(responseField, shf.getValue());  
> > > }  
> > > }  
> > > }
> > 
> > > - David
> > 
> > > On Nov 27, 3:20 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > > 
> > > > Do you rely on \_source or do you specifically store fields? If so,  
> > > > then I  
> > > > don't really understand this big difference, since when you ask for  
> > > > specific fields, it still gets loaded and parsed (with the "large"  
> > > > field),  
> > > > and if you don't, it just a matter of passing the \_source as is  
> > > > through  
> > > > the  
> > > > wire.
> > 
> > > > On Fri, Nov 25, 2011 at 1:08 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com)  
> > > > wrote:
> > > > 
> > > > > Hi Shay,
> > 
> > > > > So these are the stats for this operation:
> > 
> > > > > Timer timer = Timer.startTimer();  
> > > > > SearchResponse searchResponse = searchReq.execute().actionGet();  
> > > > > timer.endTimer("QUERY TIME ---\>\>\>");
> > 
> > > > > # Stats
> > > > > 
> > > > > 1. All fields but 'large' field (obtained by running against an old  
> > > > > index without this in):
> > 
> > > > > INFO: QUERY TIME ---\>\>\>366ms
> > 
> > > > > 1. One field - obtained from the index with the 'large' field in,  
> > > > > but  
> > > > > just asking for one field using  
> > > > > "searchReq.addFields("person\_name");"
> > 
> > > > > INFO: QUERY TIME ---\>\>\>344ms
> > 
> > > > > 1. All fields including 'large' field:
> > 
> > > > > INFO: QUERY TIME ---\>\>\>2922ms
> > 
> > > > > I think that proves that it is the serialization that is causing  
> > > > > the  
> > > > > problem, rather than the search.
> > 
> > > > > In that case, do you think the above approach will be a good way of  
> > > > > stoping that object being serilized by default?
> > 
> > > > > Or, a good alternative would be to have a method on  
> > > > > SearchRequestBuilder like:
> > 
> > > > > searchReq.removeFields("weekViewMap");
> > 
> > > > > ```
> > > > > or
> > > > > 
> > > > > ```
> > 
> > > > > searchReq.doNotSerializeFields("weekViewMap");
> > 
> > > > > For the default (fields = null), which pulls back the source.
> > 
> > > > > -David
> > 
> > > > > On Nov 24, 6:26 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > > > > 
> > > > > > You mean the object is new that you now add to your document? I  
> > > > > > suggest  
> > > > > > you  
> > > > > > first understand why its slower, for example, by asking for empty  
> > > > > > fields,  
> > > > > > or even \_source.
> > 
> > > > > > On Thu, Nov 24, 2011 at 7:08 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com)  
> > > > > > wrote:
> > > > > > 
> > > > > > > Having implemented the solution using enabled=false on a normal  
> > > > > > > object  
> > > > > > > type - I find that my query time has increased massively,  
> > > > > > > probably  
> > > > > > > because the queries I execute almost always access the "source"  
> > > > > > > object, which now has to deserialize additional very big  
> > > > > > > objects.
> > 
> > > > > > > ## From the Nested Object Documentation
> > > > > > > 
> > > > > > > "Those internal nested documents are automatically masked away  
> > > > > > > when  
> > > > > > > doing operations against the index (like searching with a  
> > > > > > > match\_all  
> > > > > > > query), and they bubble out when using the nested query."
> > 
> > > > > > > So, my idea is, to offer users the functionality to include  
> > > > > > > this  
> > > > > > > object (using a checkbox option) by pushing the large HashMap  
> > > > > > > into  
> > > > > > > a  
> > > > > > > nested property where, as I understand it, nested objects are  
> > > > > > > only  
> > > > > > > de-  
> > > > > > > serialized if a query is made against them.
> > 
> > > > > > > My Ideas is to define the mapping, as follows:
> > 
> > > > > > > {  
> > > > > > > "MainType" : {  
> > > > > > > "properties" : {  
> > > > > > > "nestedObject" : {  
> > > > > > > "type" : "nested",  
> > > > > > > "properties" : {  
> > > > > > > "constantField" :  
> > > > > > > {"type" :  
> > > > > > > "string", "index" : "not\_analyzed"},  
> > > > > > > "weekViewMap" : {"type"  
> > > > > > > :  
> > > > > > > "object",  
> > > > > > > "enabled" : false}  
> > > > > > > }  
> > > > > > > }  
> > > > > > > }  
> > > > > > > }  
> > > > > > > }
> > 
> > > > > > > So, I would "turn on" the weekView serialization, by making a  
> > > > > > > Nested  
> > > > > > > TermFilter search, where  
> > > > > > > nestedObject.contstantField="ALL-OBJECTS-HAVE-  
> > > > > > > THIS".
> > 
> > > > > > > ## Questions
> > > > > > > 
> > > > > > > 1. Will this work?
> > > > > > > 2. Will a query against another nested object also cause this  
> > > > > > > nested  
> > > > > > > object to be included in the source, or do the different nested  
> > > > > > > objects act indepently?
> > 
> > > > > > > - David
> > 
> > > > > > > On Nov 22, 8:34 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > > > > > > 
> > > > > > > > You mean not index that big "hashmap"? You can map the object  
> > > > > > > > level  
> > > > > > > > property of it with enabled set to false, which means it  
> > > > > > > > will not  
> > > > > > > > even go  
> > > > > > > > and try and map the object represented by it.
> > 
> > > > > > > > On Mon, Nov 21, 2011 at 9:34 PM, davrob2 \<  
> > > > > > > > [davirobe...@gmail.com](mailto:davirobe...@gmail.com)\>  
> > > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > Hi,
> > 
> > > > > > > > > I would like to store a large hashmap in my index without  
> > > > > > > > > mapping  
> > > > > > > > > it  
> > > > > > > > > but having available in the \_source.  
> > > > > > > > > Essentially I want to use Elasticsearch as an object store,  
> > > > > > > > > where I  
> > > > > > > > > can associate a hashmap with an index id rather than make a  
> > > > > > > > > call  
> > > > > > > > > out  
> > > > > > > > > to Mongo DB, and the like, using the id as a reference.
> > 
> > > > > > > > > -David.

---

<div class="post-metadata">

**Author:** ![davrob](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@davrob](https://discuss.elastic.co/u/davrob)\
**Post date:** [November 28, 2011, 7:15pm UTC](https://discuss.elastic.co/t/store-large-object-in-index-without-mapping/5930/12 "2011-11-28T19:15:33Z")

</div>

Hi Shay,

Do you have an example Gist of a Java based recreation, it would be  
helpful to copy.

- David.

On Nov 28, 4:27 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:

> If thats the case, then its really strange. Can you post a recreation that  
> shows this (can be Java code, with a simple "main class" repro)?
> 
> On Mon, Nov 28, 2011 at 6:10 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote:
> 
> > Hi Shay,
> 
> > Well, yes, I think your instinct is right, the difference in the files  
> > is only very small, here's an example:
> 
> > [Week View Map · GitHub](https://gist.github.com/1400795)
> 
> > The Average size of Contact document is 1327 characters, while the  
> > average size of the contacts with the added hashmap is 1699  
> > characters.
> 
> > So, I'm not sure what is causing the issue, I do have a map within the  
> > map, but I'd have thought the JSON serialization API would have no  
> > real problem with that.
> 
> > The work around I have in mind at the moment is to have a "fat" type  
> > "fatcontact" and a skinny one "contact" that are identical apart from  
> > the embedded HashMap. If users want to search on the embedded map I  
> > switch types to search on "fatcontact" type - but this is a pretty  
> > ugly work-around, because I index everything twice.
> 
> > -David.
> 
> > On Nov 28, 11:23 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > 
> > > Right, but it still gets loaded and extracted, the \_source, when you ask  
> > > for a field. And, just transferring it seems strange that it will cause  
> > > that big of a time difference. How big is the document that you index  
> > > (with  
> > > that \_map)?
> 
> > > On Mon, Nov 28, 2011 at 12:06 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote:
> > > 
> > > > Hi Shay,
> 
> > > > Yes, I do rely on \_source usually, but when I specify the field to  
> > > > search using "searchReq.addFields("person\_name");" then the source  
> > > > returned is null and I just get the individual field as below:
> 
> > > > for (SearchHit hit : queryResponse.getHits()){  
> > > > Map\<String, Object\> source = hit.getSource();  
> > > > if (source == null){  
> > > > source = new HashMap\<String, Object\>();  
> > > > for (String responseField :  
> > > > searchBean.getSearch().getResponseFields()){  
> > > > SearchHitField shf = hit.field(responseField);  
> > > > if (shf != null){  
> > > > source.put(responseField, shf.getValue());  
> > > > }  
> > > > }  
> > > > }
> 
> > > > - David
> 
> > > > On Nov 27, 3:20 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > > > 
> > > > > Do you rely on \_source or do you specifically store fields? If so,  
> > > > > then I  
> > > > > don't really understand this big difference, since when you ask for  
> > > > > specific fields, it still gets loaded and parsed (with the "large"  
> > > > > field),  
> > > > > and if you don't, it just a matter of passing the \_source as is  
> > > > > through  
> > > > > the  
> > > > > wire.
> 
> > > > > On Fri, Nov 25, 2011 at 1:08 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com)  
> > > > > wrote:
> > > > > 
> > > > > > Hi Shay,
> 
> > > > > > So these are the stats for this operation:
> 
> > > > > > Timer timer = Timer.startTimer();  
> > > > > > SearchResponse searchResponse = searchReq.execute().actionGet();  
> > > > > > timer.endTimer("QUERY TIME ---\>\>\>");
> 
> > > > > > # Stats
> > > > > > 
> > > > > > 1. All fields but 'large' field (obtained by running against an old  
> > > > > > index without this in):
> 
> > > > > > INFO: QUERY TIME ---\>\>\>366ms
> 
> > > > > > 1. One field - obtained from the index with the 'large' field in,  
> > > > > > but  
> > > > > > just asking for one field using  
> > > > > > "searchReq.addFields("person\_name");"
> 
> > > > > > INFO: QUERY TIME ---\>\>\>344ms
> 
> > > > > > 1. All fields including 'large' field:
> 
> > > > > > INFO: QUERY TIME ---\>\>\>2922ms
> 
> > > > > > I think that proves that it is the serialization that is causing  
> > > > > > the  
> > > > > > problem, rather than the search.
> 
> > > > > > In that case, do you think the above approach will be a good way of  
> > > > > > stoping that object being serilized by default?
> 
> > > > > > Or, a good alternative would be to have a method on  
> > > > > > SearchRequestBuilder like:
> 
> > > > > > searchReq.removeFields("weekViewMap");
> 
> > > > > > ```
> > > > > > or
> > > > > > 
> > > > > > ```
> 
> > > > > > searchReq.doNotSerializeFields("weekViewMap");
> 
> > > > > > For the default (fields = null), which pulls back the source.
> 
> > > > > > -David
> 
> > > > > > On Nov 24, 6:26 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > > > > > 
> > > > > > > You mean the object is new that you now add to your document? I  
> > > > > > > suggest  
> > > > > > > you  
> > > > > > > first understand why its slower, for example, by asking for empty  
> > > > > > > fields,  
> > > > > > > or even \_source.
> 
> > > > > > > On Thu, Nov 24, 2011 at 7:08 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com)  
> > > > > > > wrote:
> > > > > > > 
> > > > > > > > Having implemented the solution using enabled=false on a normal  
> > > > > > > > object  
> > > > > > > > type - I find that my query time has increased massively,  
> > > > > > > > probably  
> > > > > > > > because the queries I execute almost always access the "source"  
> > > > > > > > object, which now has to deserialize additional very big  
> > > > > > > > objects.
> 
> > > > > > > > ## From the Nested Object Documentation
> > > > > > > > 
> > > > > > > > "Those internal nested documents are automatically masked away  
> > > > > > > > when  
> > > > > > > > doing operations against the index (like searching with a  
> > > > > > > > match\_all  
> > > > > > > > query), and they bubble out when using the nested query."
> 
> > > > > > > > So, my idea is, to offer users the functionality to include  
> > > > > > > > this  
> > > > > > > > object (using a checkbox option) by pushing the large HashMap  
> > > > > > > > into  
> > > > > > > > a  
> > > > > > > > nested property where, as I understand it, nested objects are  
> > > > > > > > only  
> > > > > > > > de-  
> > > > > > > > serialized if a query is made against them.
> 
> > > > > > > > My Ideas is to define the mapping, as follows:
> 
> > > > > > > > {  
> > > > > > > > "MainType" : {  
> > > > > > > > "properties" : {  
> > > > > > > > "nestedObject" : {  
> > > > > > > > "type" : "nested",  
> > > > > > > > "properties" : {  
> > > > > > > > "constantField" :  
> > > > > > > > {"type" :  
> > > > > > > > "string", "index" : "not\_analyzed"},  
> > > > > > > > "weekViewMap" : {"type"  
> > > > > > > > :  
> > > > > > > > "object",  
> > > > > > > > "enabled" : false}  
> > > > > > > > }  
> > > > > > > > }  
> > > > > > > > }  
> > > > > > > > }  
> > > > > > > > }
> 
> > > > > > > > So, I would "turn on" the weekView serialization, by making a  
> > > > > > > > Nested  
> > > > > > > > TermFilter search, where  
> > > > > > > > nestedObject.contstantField="ALL-OBJECTS-HAVE-  
> > > > > > > > THIS".
> 
> > > > > > > > ## Questions
> > > > > > > > 
> > > > > > > > 1. Will this work?
> > > > > > > > 2. Will a query against another nested object also cause this  
> > > > > > > > nested  
> > > > > > > > object to be included in the source, or do the different nested  
> > > > > > > > objects act indepently?
> 
> > > > > > > > - David
> 
> > > > > > > > On Nov 22, 8:34 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > > > > > > > 
> > > > > > > > > You mean not index that big "hashmap"? You can map the object  
> > > > > > > > > level  
> > > > > > > > > property of it with enabled set to false, which means it  
> > > > > > > > > will not  
> > > > > > > > > even go  
> > > > > > > > > and try and map the object represented by it.
> 
> > > > > > > > > On Mon, Nov 21, 2011 at 9:34 PM, davrob2 \<  
> > > > > > > > > [davirobe...@gmail.com](mailto:davirobe...@gmail.com)\>  
> > > > > > > > > wrote:
> > > > > > > > > 
> > > > > > > > > > Hi,
> 
> > > > > > > > > > I would like to store a large hashmap in my index without  
> > > > > > > > > > mapping  
> > > > > > > > > > it  
> > > > > > > > > > but having available in the \_source.  
> > > > > > > > > > Essentially I want to use Elasticsearch as an object store,  
> > > > > > > > > > where I  
> > > > > > > > > > can associate a hashmap with an index id rather than make a  
> > > > > > > > > > call  
> > > > > > > > > > out  
> > > > > > > > > > to Mongo DB, and the like, using the id as a reference.
> 
> > > > > > > > > > -David.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [November 28, 2011, 8:48pm UTC](https://discuss.elastic.co/t/store-large-object-in-index-without-mapping/5930/13 "2011-11-28T20:48:45Z")

</div>

Not one offhand, but it should be simple. Create a simple main class, start  
a client in it, index a sample doc(s), and then execute the search requests  
that show the problem, where one takes considerable more time than the  
others.

On Mon, Nov 28, 2011 at 9:15 PM, davrob2 [daviroberts@gmail.com](mailto:daviroberts@gmail.com) wrote:

> Hi Shay,
> 
> Do you have an example Gist of a Java based recreation, it would be  
> helpful to copy.
> 
> - David.
> 
> On Nov 28, 4:27 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> 
> > If thats the case, then its really strange. Can you post a recreation  
> > that  
> > shows this (can be Java code, with a simple "main class" repro)?
> > 
> > On Mon, Nov 28, 2011 at 6:10 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote:
> > 
> > > Hi Shay,
> > 
> > > Well, yes, I think your instinct is right, the difference in the files  
> > > is only very small, here's an example:
> > 
> > > [Week View Map · GitHub](https://gist.github.com/1400795)
> > 
> > > The Average size of Contact document is 1327 characters, while the  
> > > average size of the contacts with the added hashmap is 1699  
> > > characters.
> > 
> > > So, I'm not sure what is causing the issue, I do have a map within the  
> > > map, but I'd have thought the JSON serialization API would have no  
> > > real problem with that.
> > 
> > > The work around I have in mind at the moment is to have a "fat" type  
> > > "fatcontact" and a skinny one "contact" that are identical apart from  
> > > the embedded HashMap. If users want to search on the embedded map I  
> > > switch types to search on "fatcontact" type - but this is a pretty  
> > > ugly work-around, because I index everything twice.
> > 
> > > -David.
> > 
> > > On Nov 28, 11:23 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > > 
> > > > Right, but it still gets loaded and extracted, the \_source, when you  
> > > > ask  
> > > > for a field. And, just transferring it seems strange that it will  
> > > > cause  
> > > > that big of a time difference. How big is the document that you index  
> > > > (with  
> > > > that \_map)?
> > 
> > > > On Mon, Nov 28, 2011 at 12:06 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com)  
> > > > wrote:
> > > > 
> > > > > Hi Shay,
> > 
> > > > > Yes, I do rely on \_source usually, but when I specify the field to  
> > > > > search using "searchReq.addFields("person\_name");" then the source  
> > > > > returned is null and I just get the individual field as below:
> > 
> > > > > for (SearchHit hit : queryResponse.getHits()){  
> > > > > Map\<String, Object\> source = hit.getSource();  
> > > > > if (source == null){  
> > > > > source = new HashMap\<String, Object\>();  
> > > > > for (String responseField :  
> > > > > searchBean.getSearch().getResponseFields()){  
> > > > > SearchHitField shf = hit.field(responseField);  
> > > > > if (shf != null){  
> > > > > source.put(responseField, shf.getValue());  
> > > > > }  
> > > > > }  
> > > > > }
> > 
> > > > > - David
> > 
> > > > > On Nov 27, 3:20 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > > > > 
> > > > > > Do you rely on \_source or do you specifically store fields? If  
> > > > > > so,  
> > > > > > then I  
> > > > > > don't really understand this big difference, since when you ask  
> > > > > > for  
> > > > > > specific fields, it still gets loaded and parsed (with the  
> > > > > > "large"  
> > > > > > field),  
> > > > > > and if you don't, it just a matter of passing the \_source as is  
> > > > > > through  
> > > > > > the  
> > > > > > wire.
> > 
> > > > > > On Fri, Nov 25, 2011 at 1:08 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com)  
> > > > > > wrote:
> > > > > > 
> > > > > > > Hi Shay,
> > 
> > > > > > > So these are the stats for this operation:
> > 
> > > > > > > Timer timer = Timer.startTimer();  
> > > > > > > SearchResponse searchResponse =  
> > > > > > > searchReq.execute().actionGet();  
> > > > > > > timer.endTimer("QUERY TIME ---\>\>\>");
> > 
> > > > > > > # Stats
> > > > > > > 
> > > > > > > 1. All fields but 'large' field (obtained by running against  
> > > > > > > an old  
> > > > > > > index without this in):
> > 
> > > > > > > INFO: QUERY TIME ---\>\>\>366ms
> > 
> > > > > > > 1. One field - obtained from the index with the 'large' field  
> > > > > > > in,  
> > > > > > > but  
> > > > > > > just asking for one field using  
> > > > > > > "searchReq.addFields("person\_name");"
> > 
> > > > > > > INFO: QUERY TIME ---\>\>\>344ms
> > 
> > > > > > > 1. All fields including 'large' field:
> > 
> > > > > > > INFO: QUERY TIME ---\>\>\>2922ms
> > 
> > > > > > > I think that proves that it is the serialization that is  
> > > > > > > causing  
> > > > > > > the  
> > > > > > > problem, rather than the search.
> > 
> > > > > > > In that case, do you think the above approach will be a good  
> > > > > > > way of  
> > > > > > > stoping that object being serilized by default?
> > 
> > > > > > > Or, a good alternative would be to have a method on  
> > > > > > > SearchRequestBuilder like:
> > 
> > > > > > > searchReq.removeFields("weekViewMap");
> > 
> > > > > > > ```
> > > > > > > or
> > > > > > > 
> > > > > > > ```
> > 
> > > > > > > searchReq.doNotSerializeFields("weekViewMap");
> > 
> > > > > > > For the default (fields = null), which pulls back the source.
> > 
> > > > > > > -David
> > 
> > > > > > > On Nov 24, 6:26 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > > > > > > 
> > > > > > > > You mean the object is new that you now add to your  
> > > > > > > > document? I  
> > > > > > > > suggest  
> > > > > > > > you  
> > > > > > > > first understand why its slower, for example, by asking for  
> > > > > > > > empty  
> > > > > > > > fields,  
> > > > > > > > or even \_source.
> > 
> > > > > > > > On Thu, Nov 24, 2011 at 7:08 PM, davrob2 \<  
> > > > > > > > [davirobe...@gmail.com](mailto:davirobe...@gmail.com)\>  
> > > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > Having implemented the solution using enabled=false on a  
> > > > > > > > > normal  
> > > > > > > > > object  
> > > > > > > > > type - I find that my query time has increased massively,  
> > > > > > > > > probably  
> > > > > > > > > because the queries I execute almost always access the  
> > > > > > > > > "source"  
> > > > > > > > > object, which now has to deserialize additional very big  
> > > > > > > > > objects.
> > 
> > > > > > > > > ## From the Nested Object Documentation
> > > > > > > > > 
> > > > > > > > > "Those internal nested documents are automatically masked  
> > > > > > > > > away  
> > > > > > > > > when  
> > > > > > > > > doing operations against the index (like searching with a  
> > > > > > > > > match\_all  
> > > > > > > > > query), and they bubble out when using the nested query."
> > 
> > > > > > > > > So, my idea is, to offer users the functionality to include  
> > > > > > > > > this  
> > > > > > > > > object (using a checkbox option) by pushing the large  
> > > > > > > > > HashMap  
> > > > > > > > > into  
> > > > > > > > > a  
> > > > > > > > > nested property where, as I understand it, nested objects  
> > > > > > > > > are  
> > > > > > > > > only  
> > > > > > > > > de-  
> > > > > > > > > serialized if a query is made against them.
> > 
> > > > > > > > > My Ideas is to define the mapping, as follows:
> > 
> > > > > > > > > {  
> > > > > > > > > "MainType" : {  
> > > > > > > > > "properties" : {  
> > > > > > > > > "nestedObject" : {  
> > > > > > > > > "type" : "nested",  
> > > > > > > > > "properties" : {  
> > > > > > > > > "constantField" :  
> > > > > > > > > {"type" :  
> > > > > > > > > "string", "index" : "not\_analyzed"},  
> > > > > > > > > "weekViewMap" :  
> > > > > > > > > {"type"  
> > > > > > > > > :  
> > > > > > > > > "object",  
> > > > > > > > > "enabled" : false}  
> > > > > > > > > }  
> > > > > > > > > }  
> > > > > > > > > }  
> > > > > > > > > }  
> > > > > > > > > }
> > 
> > > > > > > > > So, I would "turn on" the weekView serialization, by  
> > > > > > > > > making a  
> > > > > > > > > Nested  
> > > > > > > > > TermFilter search, where  
> > > > > > > > > nestedObject.contstantField="ALL-OBJECTS-HAVE-  
> > > > > > > > > THIS".
> > 
> > > > > > > > > ## Questions
> > > > > > > > > 
> > > > > > > > > 1. Will this work?
> > > > > > > > > 2. Will a query against another nested object also cause  
> > > > > > > > > this  
> > > > > > > > > nested  
> > > > > > > > > object to be included in the source, or do the different  
> > > > > > > > > nested  
> > > > > > > > > objects act indepently?
> > 
> > > > > > > > > - David
> > 
> > > > > > > > > On Nov 22, 8:34 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > > > > > > > > 
> > > > > > > > > > You mean not index that big "hashmap"? You can map the  
> > > > > > > > > > object  
> > > > > > > > > > level  
> > > > > > > > > > property of it with enabled set to false, which means it  
> > > > > > > > > > will not  
> > > > > > > > > > even go  
> > > > > > > > > > and try and map the object represented by it.
> > 
> > > > > > > > > > On Mon, Nov 21, 2011 at 9:34 PM, davrob2 \<  
> > > > > > > > > > [davirobe...@gmail.com](mailto:davirobe...@gmail.com)\>  
> > > > > > > > > > wrote:
> > > > > > > > > > 
> > > > > > > > > > > Hi,
> > 
> > > > > > > > > > > I would like to store a large hashmap in my index  
> > > > > > > > > > > without  
> > > > > > > > > > > mapping  
> > > > > > > > > > > it  
> > > > > > > > > > > but having available in the \_source.  
> > > > > > > > > > > Essentially I want to use Elasticsearch as an object  
> > > > > > > > > > > store,  
> > > > > > > > > > > where I  
> > > > > > > > > > > can associate a hashmap with an index id rather than  
> > > > > > > > > > > make a  
> > > > > > > > > > > call  
> > > > > > > > > > > out  
> > > > > > > > > > > to Mongo DB, and the like, using the id as a reference.
> > 
> > > > > > > > > > > -David.

---

<div class="post-metadata">

**Author:** ![davrob](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@davrob](https://discuss.elastic.co/u/davrob)\
**Post date:** [November 30, 2011, 2:29pm UTC](https://discuss.elastic.co/t/store-large-object-in-index-without-mapping/5930/14 "2011-11-30T14:29:19Z")

</div>

Hi Shay,

So I ran my recreation here: [Base Recreation Class and Test Class · GitHub](https://gist.github.com/1409236)

And its entirely my own stupidity - I was measuring the network  
latency to New York from my desktop, as opposed to running the client  
and server both in New York, or both locally.

Thanks for your patience, hopefully other java users will find the  
BaseClass in the recreation usefull for their own work.

- David.

On Nov 28, 8:48 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:

> Not one offhand, but it should be simple. Create a simple main class, start  
> a client in it, index a sample doc(s), and then execute the search requests  
> that show the problem, where one takes considerable more time than the  
> others.
> 
> On Mon, Nov 28, 2011 at 9:15 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote:
> 
> > Hi Shay,
> 
> > Do you have an example Gist of a Java based recreation, it would be  
> > helpful to copy.
> 
> > - David.
> 
> > On Nov 28, 4:27 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > 
> > > If thats the case, then its really strange. Can you post a recreation  
> > > that  
> > > shows this (can be Java code, with a simple "main class" repro)?
> 
> > > On Mon, Nov 28, 2011 at 6:10 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote:
> > > 
> > > > Hi Shay,
> 
> > > > Well, yes, I think your instinct is right, the difference in the files  
> > > > is only very small, here's an example:
> 
> > > > [Week View Map · GitHub](https://gist.github.com/1400795)
> 
> > > > The Average size of Contact document is 1327 characters, while the  
> > > > average size of the contacts with the added hashmap is 1699  
> > > > characters.
> 
> > > > So, I'm not sure what is causing the issue, I do have a map within the  
> > > > map, but I'd have thought the JSON serialization API would have no  
> > > > real problem with that.
> 
> > > > The work around I have in mind at the moment is to have a "fat" type  
> > > > "fatcontact" and a skinny one "contact" that are identical apart from  
> > > > the embedded HashMap. If users want to search on the embedded map I  
> > > > switch types to search on "fatcontact" type - but this is a pretty  
> > > > ugly work-around, because I index everything twice.
> 
> > > > -David.
> 
> > > > On Nov 28, 11:23 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > > > 
> > > > > Right, but it still gets loaded and extracted, the \_source, when you  
> > > > > ask  
> > > > > for a field. And, just transferring it seems strange that it will  
> > > > > cause  
> > > > > that big of a time difference. How big is the document that you index  
> > > > > (with  
> > > > > that \_map)?
> 
> > > > > On Mon, Nov 28, 2011 at 12:06 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com)  
> > > > > wrote:
> > > > > 
> > > > > > Hi Shay,
> 
> > > > > > Yes, I do rely on \_source usually, but when I specify the field to  
> > > > > > search using "searchReq.addFields("person\_name");" then the source  
> > > > > > returned is null and I just get the individual field as below:
> 
> > > > > > for (SearchHit hit : queryResponse.getHits()){  
> > > > > > Map\<String, Object\> source = hit.getSource();  
> > > > > > if (source == null){  
> > > > > > source = new HashMap\<String, Object\>();  
> > > > > > for (String responseField :  
> > > > > > searchBean.getSearch().getResponseFields()){  
> > > > > > SearchHitField shf = hit.field(responseField);  
> > > > > > if (shf != null){  
> > > > > > source.put(responseField, shf.getValue());  
> > > > > > }  
> > > > > > }  
> > > > > > }
> 
> > > > > > - David
> 
> > > > > > On Nov 27, 3:20 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > > > > > 
> > > > > > > Do you rely on \_source or do you specifically store fields? If  
> > > > > > > so,  
> > > > > > > then I  
> > > > > > > don't really understand this big difference, since when you ask  
> > > > > > > for  
> > > > > > > specific fields, it still gets loaded and parsed (with the  
> > > > > > > "large"  
> > > > > > > field),  
> > > > > > > and if you don't, it just a matter of passing the \_source as is  
> > > > > > > through  
> > > > > > > the  
> > > > > > > wire.
> 
> > > > > > > On Fri, Nov 25, 2011 at 1:08 PM, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com)  
> > > > > > > wrote:
> > > > > > > 
> > > > > > > > Hi Shay,
> 
> > > > > > > > So these are the stats for this operation:
> 
> > > > > > > > Timer timer = Timer.startTimer();  
> > > > > > > > SearchResponse searchResponse =  
> > > > > > > > searchReq.execute().actionGet();  
> > > > > > > > timer.endTimer("QUERY TIME ---\>\>\>");
> 
> > > > > > > > # Stats
> > > > > > > > 
> > > > > > > > 1. All fields but 'large' field (obtained by running against  
> > > > > > > > an old  
> > > > > > > > index without this in):
> 
> > > > > > > > INFO: QUERY TIME ---\>\>\>366ms
> 
> > > > > > > > 1. One field - obtained from the index with the 'large' field  
> > > > > > > > in,  
> > > > > > > > but  
> > > > > > > > just asking for one field using  
> > > > > > > > "searchReq.addFields("person\_name");"
> 
> > > > > > > > INFO: QUERY TIME ---\>\>\>344ms
> 
> > > > > > > > 1. All fields including 'large' field:
> 
> > > > > > > > INFO: QUERY TIME ---\>\>\>2922ms
> 
> > > > > > > > I think that proves that it is the serialization that is  
> > > > > > > > causing  
> > > > > > > > the  
> > > > > > > > problem, rather than the search.
> 
> > > > > > > > In that case, do you think the above approach will be a good  
> > > > > > > > way of  
> > > > > > > > stoping that object being serilized by default?
> 
> > > > > > > > Or, a good alternative would be to have a method on  
> > > > > > > > SearchRequestBuilder like:
> 
> > > > > > > > searchReq.removeFields("weekViewMap");
> 
> > > > > > > > ```
> > > > > > > > or
> > > > > > > > 
> > > > > > > > ```
> 
> > > > > > > > searchReq.doNotSerializeFields("weekViewMap");
> 
> > > > > > > > For the default (fields = null), which pulls back the source.
> 
> > > > > > > > -David
> 
> > > > > > > > On Nov 24, 6:26 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > > > > > > > 
> > > > > > > > > You mean the object is new that you now add to your  
> > > > > > > > > document? I  
> > > > > > > > > suggest  
> > > > > > > > > you  
> > > > > > > > > first understand why its slower, for example, by asking for  
> > > > > > > > > empty  
> > > > > > > > > fields,  
> > > > > > > > > or even \_source.
> 
> > > > > > > > > On Thu, Nov 24, 2011 at 7:08 PM, davrob2 \<  
> > > > > > > > > [davirobe...@gmail.com](mailto:davirobe...@gmail.com)\>  
> > > > > > > > > wrote:
> > > > > > > > > 
> > > > > > > > > > Having implemented the solution using enabled=false on a  
> > > > > > > > > > normal  
> > > > > > > > > > object  
> > > > > > > > > > type - I find that my query time has increased massively,  
> > > > > > > > > > probably  
> > > > > > > > > > because the queries I execute almost always access the  
> > > > > > > > > > "source"  
> > > > > > > > > > object, which now has to deserialize additional very big  
> > > > > > > > > > objects.
> 
> > > > > > > > > > ## From the Nested Object Documentation
> > > > > > > > > > 
> > > > > > > > > > "Those internal nested documents are automatically masked  
> > > > > > > > > > away  
> > > > > > > > > > when  
> > > > > > > > > > doing operations against the index (like searching with a  
> > > > > > > > > > match\_all  
> > > > > > > > > > query), and they bubble out when using the nested query."
> 
> > > > > > > > > > So, my idea is, to offer users the functionality to include  
> > > > > > > > > > this  
> > > > > > > > > > object (using a checkbox option) by pushing the large  
> > > > > > > > > > HashMap  
> > > > > > > > > > into  
> > > > > > > > > > a  
> > > > > > > > > > nested property where, as I understand it, nested objects  
> > > > > > > > > > are  
> > > > > > > > > > only  
> > > > > > > > > > de-  
> > > > > > > > > > serialized if a query is made against them.
> 
> > > > > > > > > > My Ideas is to define the mapping, as follows:
> 
> > > > > > > > > > {  
> > > > > > > > > > "MainType" : {  
> > > > > > > > > > "properties" : {  
> > > > > > > > > > "nestedObject" : {  
> > > > > > > > > > "type" : "nested",  
> > > > > > > > > > "properties" : {  
> > > > > > > > > > "constantField" :  
> > > > > > > > > > {"type" :  
> > > > > > > > > > "string", "index" : "not\_analyzed"},  
> > > > > > > > > > "weekViewMap" :  
> > > > > > > > > > {"type"  
> > > > > > > > > > :  
> > > > > > > > > > "object",  
> > > > > > > > > > "enabled" : false}  
> > > > > > > > > > }  
> > > > > > > > > > }  
> > > > > > > > > > }  
> > > > > > > > > > }  
> > > > > > > > > > }
> 
> > > > > > > > > > So, I would "turn on" the weekView serialization, by  
> > > > > > > > > > making a  
> > > > > > > > > > Nested  
> > > > > > > > > > TermFilter search, where  
> > > > > > > > > > nestedObject.contstantField="ALL-OBJECTS-HAVE-  
> > > > > > > > > > THIS".
> 
> > > > > > > > > > ## Questions
> > > > > > > > > > 
> > > > > > > > > > 1. Will this work?
> > > > > > > > > > 2. Will a query against another nested object also cause  
> > > > > > > > > > this  
> > > > > > > > > > nested  
> > > > > > > > > > object to be included in the source, or do the different  
> > > > > > > > > > nested  
> > > > > > > > > > objects act indepently?
> 
> > > > > > > > > > - David
> 
> > > > > > > > > > On Nov 22, 8:34 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> > > > > > > > > > 
> > > > > > > > > > > You mean not index that big "hashmap"? You can map the  
> > > > > > > > > > > object  
> > > > > > > > > > > level  
> > > > > > > > > > > property of it with enabled set to false, which means it  
> > > > > > > > > > > will not  
> > > > > > > > > > > even go  
> > > > > > > > > > > and try and map the object represented by it.
> 
> > > > > > > > > > > On Mon, Nov 21, 2011 at 9:34 PM, davrob2 \<  
> > > > > > > > > > > [davirobe...@gmail.com](mailto:davirobe...@gmail.com)\>  
> > > > > > > > > > > wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > Hi,
> 
> > > > > > > > > > > > I would like to store a large hashmap in my index  
> > > > > > > > > > > > without  
> > > > > > > > > > > > mapping  
> > > > > > > > > > > > it  
> > > > > > > > > > > > but having available in the \_source.  
> > > > > > > > > > > > Essentially I want to use Elasticsearch as an object  
> > > > > > > > > > > > store,  
> > > > > > > > > > > > where I  
> > > > > > > > > > > > can associate a hashmap with an index id rather than  
> > > > > > > > > > > > make a  
> > > > > > > > > > > > call  
> > > > > > > > > > > > out  
> > > > > > > > > > > > to Mongo DB, and the like, using the id as a reference.
> 
> > > > > > > > > > > > -David.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:46am UTC](https://discuss.elastic.co/t/store-large-object-in-index-without-mapping/5930/15 "2017-07-06T03:46:56Z")

</div>