# Partial\_fields: streaming or in-memory?

**URL:** <https://discuss.elastic.co/t/partial-fields-streaming-or-in-memory/11739>\
**Category:** Elasticsearch\
**Created:** [April 29, 2013, 11:39pm UTC](https://discuss.elastic.co/t/partial-fields-streaming-or-in-memory/11739 "2013-04-29T23:39:02Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![btiernay](https://avatars.discourse-cdn.com/v4/letter/b/bb73d2/32.png) [@btiernay](https://discuss.elastic.co/u/btiernay)\
**Post date:** [April 29, 2013, 11:39pm UTC](https://discuss.elastic.co/t/partial-fields-streaming-or-in-memory/11739/1 "2013-04-29T23:39:02Z")

</div>

Hi ES'ers!

Just curious if anyone knows if partial\_fields  
[http://www.elasticsearch.org/guide/reference/api/search/fields/](http://www.elasticsearch.org/guide/reference/api/search/fields/)uses a  
streaming approach to filter \_source[http://www.elasticsearch.org/guide/reference/mapping/source-field/](http://www.elasticsearch.org/guide/reference/mapping/source-field/) documents.  
The reason I ask is that we index some relatively large documents and if ES  
can perform streaming filtering (as I believe Jackson supports) this would  
be ideal. I couldn't really figure this out by browsing the source. It  
looks as though this is implemented in-memory by pruning map keys. I could  
be wrong 🙂

Thanks again!

Bob

References:

[1] [https://github.com/elasticsearch/elasticsearch/issues/1570](https://github.com/elasticsearch/elasticsearch/issues/1570)  
[2]  
[https://github.com/elasticsearch/elasticsearch/commit/cc3f44473fb7e0e543707403cea87e9b7ca7ad57](https://github.com/elasticsearch/elasticsearch/commit/cc3f44473fb7e0e543707403cea87e9b7ca7ad57)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![btiernay](https://avatars.discourse-cdn.com/v4/letter/b/bb73d2/32.png) [@btiernay](https://discuss.elastic.co/u/btiernay)\
**Post date:** [May 7, 2013, 1:44am UTC](https://discuss.elastic.co/t/partial-fields-streaming-or-in-memory/11739/2 "2013-05-07T01:44:30Z")

</div>

Doing a bit more digging it looks as though this is the case:

[1] hitExecute  
calls  
[1] Object value = context.lookup().source().filter(field.includes(), field.  
excludes());  
calls  
[2] loadSourceIfNeeded  
calls  
[2] reader.document(docId, sourceFieldVisitor);  
calls  
[3]

[1]

> <https://github.com/elastic/elasticsearch/blob/c93babed42f1020f7e348808bec4182fc109c030/src/main/java/org/elasticsearch/search/fetch/partial/PartialFieldsFetchSubPhase.java>

[2]

> <https://github.com/elastic/elasticsearch/blob/c93babed42f1020f7e348808bec4182fc109c030/src/main/java/org/elasticsearch/search/lookup/SourceLookup.java>

[3] [IndexReader (Lucene 4.0.0 API)](http://lucene.apache.org/core/4_0_0/core/org/apache/lucene/index/IndexReader.html#document)(int,  
org.apache.lucene.index.StoredFieldVisitor)

On Monday, 29 April 2013 19:39:02 UTC-4, btiernay wrote:

> Hi ES'ers!
> 
> Just curious if anyone knows if partial\_fields  
> [http://www.elasticsearch.org/guide/reference/api/search/fields/](http://www.elasticsearch.org/guide/reference/api/search/fields/)uses a  
> streaming approach to filter \_source[http://www.elasticsearch.org/guide/reference/mapping/source-field/](http://www.elasticsearch.org/guide/reference/mapping/source-field/) documents.  
> The reason I ask is that we index some relatively large documents and if ES  
> can perform streaming filtering (as I believe Jackson supports) this would  
> be ideal. I couldn't really figure this out by browsing the source. It  
> looks as though this is implemented in-memory by pruning map keys. I could  
> be wrong 🙂
> 
> Thanks again!
> 
> Bob
> 
> References:
> 
> [1] [Search: Support partial fields that can returns partial view of the \_source · Issue #1570 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1570)  
> [2]  
> [Search: Support partial fields that can returns partial view of the \_… · elastic/elasticsearch@cc3f444 · GitHub](https://github.com/elasticsearch/elasticsearch/commit/cc3f44473fb7e0e543707403cea87e9b7ca7ad57)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:38am UTC](https://discuss.elastic.co/t/partial-fields-streaming-or-in-memory/11739/3 "2017-07-06T02:38:02Z")

</div>

Closed on July 6, 2017, 2:38am UTC