Partial_fields: streaming or in-memory?

btiernay · April 29, 2013, 11:39pm

Hi ES'ers!

Just curious if anyone knows if partial_fields
http://www.elasticsearch.org/guide/reference/api/search/fields/uses a
streaming approach to filter _sourcehttp://www.elasticsearch.org/guide/reference/mapping/source-field/ documents.
The reason I ask is that we index some relatively large documents and if ES
can perform streaming filtering (as I believe Jackson supports) this would
be ideal. I couldn't really figure this out by browsing the source. It
looks as though this is implemented in-memory by pruning map keys. I could
be wrong

Thanks again!

Bob

References:

[1] https://github.com/elasticsearch/elasticsearch/issues/1570
[2]
https://github.com/elasticsearch/elasticsearch/commit/cc3f44473fb7e0e543707403cea87e9b7ca7ad57

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

btiernay · May 7, 2013, 1:44am

Doing a bit more digging it looks as though this is the case:

[1] hitExecute
calls
[1] Object value = context.lookup().source().filter(field.includes(), field.
excludes());
calls
[2] loadSourceIfNeeded
calls
[2] reader.document(docId, sourceFieldVisitor);
calls
[3]

[1]

github.com

elastic/elasticsearch/blob/c93babed42f1020f7e348808bec4182fc109c030/src/main/java/org/elasticsearch/search/fetch/partial/PartialFieldsFetchSubPhase.java

/*
 * Licensed to ElasticSearch and Shay Banon under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership. ElasticSearch licenses this
 * file to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing,
 * software distributed under the License is distributed on an
 * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 * KIND, either express or implied.  See the License for the
 * specific language governing permissions and limitations
 * under the License.
 */

package org.elasticsearch.search.fetch.partial;

This file has been truncated. show original

[2]

github.com

elastic/elasticsearch/blob/c93babed42f1020f7e348808bec4182fc109c030/src/main/java/org/elasticsearch/search/lookup/SourceLookup.java

/*
 * Licensed to Elastic Search and Shay Banon under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership. Elastic Search licenses this
 * file to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing,
 * software distributed under the License is distributed on an
 * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 * KIND, either express or implied.  See the License for the
 * specific language governing permissions and limitations
 * under the License.
 */

package org.elasticsearch.search.lookup;

This file has been truncated. show original

[3] IndexReader (Lucene 4.0.0 API)(int,
org.apache.lucene.index.StoredFieldVisitor)

On Monday, 29 April 2013 19:39:02 UTC-4, btiernay wrote:

Hi ES'ers!

Just curious if anyone knows if partial_fields
http://www.elasticsearch.org/guide/reference/api/search/fields/uses a
streaming approach to filter _sourcehttp://www.elasticsearch.org/guide/reference/mapping/source-field/ documents.
The reason I ask is that we index some relatively large documents and if ES
can perform streaming filtering (as I believe Jackson supports) this would
be ideal. I couldn't really figure this out by browsing the source. It
looks as though this is implemented in-memory by pruning map keys. I could
be wrong

Thanks again!

Bob

References:

[1] Search: Support partial fields that can returns partial view of the _source · Issue #1570 · elastic/elasticsearch · GitHub
[2]
Search: Support partial fields that can returns partial view of the _… · elastic/elasticsearch@cc3f444 · GitHub

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.