# Large (stored) fields, json source and highlighting

**URL:** <https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213>\
**Category:** Elasticsearch\
**Created:** [August 12, 2010, 3:06pm UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213 "2010-08-12T15:06:31Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tomislav\_Poljak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomislav_poljak/32/1177_2.png) [@Tomislav\_Poljak](https://discuss.elastic.co/u/Tomislav_Poljak)\
**Post date:** [August 12, 2010, 3:06pm UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213/1 "2010-08-12T15:06:31Z")

</div>

Hi,  
I really like all the features stored json (enabled \_source) provides in  
both Java API (used it in indexing/updating) and REST API (used it for  
searching).

I do however have one possible request for improvement regarding  
documents with large textual fields and overall highlighting.

When there is a requirement to index/search documents with large textual  
fields (like 'content' with text in Mb, which is not unusual), returning  
a whole json for each result in result set can be impossible (if each  
json document has a few Mb in 'content' returning 30-50 results to a  
client doesn't sound realistic or even possible in acceptable time with  
usual 'Internet' bandwidth).

But, usually it's acceptable (or even requirement) to display/return  
only highlighting snippets for 30-50 results matches and retrieve whole  
document (json source) only for a single document (when requested by  
exact ID).

To be able to provide highlighting snippets for large textual  
('content') field, it needs to be stored.

Now we are in situation where because textual field is too big (makes  
impassible to return json source for 30-50 results to a client) we need  
to store it twice in index (once as a part of original json source in  
\_source and second time as a 'content' field "store" : "yes" for  
highlighting). This make index a lot bigger.

Also, if there is a requirement to display highlighting for all fields  
(separate highlight snippets for each field where match occurred,  
without mixing fields snippets -\> stored \_all field can not be used for  
highlighting in such case) then whole document (all fields) needs to be  
stored twice.

In this case seems only logical to disable \_source field (since all  
fields are stored anyway) and when whole document needs to be retrieved  
use (newly added) fields=\* feature (I've read similar discussion thread  
which led to this enhancement)

Here is my question/proposal:

would it be possible to enable use of json \_source field for 'field  
specific' highlighting, where matching snippet needs to be returned  
separately for each field name?

Maybe to have term\_vector for each field, but to somehow 'adjust' or  
recalculate positions\_offsets to point to text snippet in \_source  
instead of stored field?

I know this is not a simple requirement, but if 'field specific'  
highlighting could somehow use stored json instead of requiring an  
individual field to be (separately) stored, that would make a great  
use/reuse of stored json \_source (and no one would ever think of  
disabling it 🙂

Tomislav

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 12, 2010, 3:53pm UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213/2 "2010-08-12T15:53:17Z")

</div>

Agreed. A bit tricky to implement, but possible. Also, note that this will  
require loading the full json from the index, and parse it in order to get  
the relevant parts from it. It won't be returned, but still loaded. So, it  
might not make sense when you have several big fields in the index, and you  
want to get fragments for one of them. But does make sense when having one  
big field.

Also, if this is implemented, it should also be possible to get specific  
fields out of the json as a response as well (similar to asking for specific  
fields in the search request, maybe call them source\_fileds).

Open an issue for this?

-shay.banon

On Thu, Aug 12, 2010 at 6:06 PM, Tomislav Poljak [tpoljak@gmail.com](mailto:tpoljak@gmail.com) wrote:

> Hi,  
> I really like all the features stored json (enabled \_source) provides in  
> both Java API (used it in indexing/updating) and REST API (used it for  
> searching).
> 
> I do however have one possible request for improvement regarding  
> documents with large textual fields and overall highlighting.
> 
> When there is a requirement to index/search documents with large textual  
> fields (like 'content' with text in Mb, which is not unusual), returning  
> a whole json for each result in result set can be impossible (if each  
> json document has a few Mb in 'content' returning 30-50 results to a  
> client doesn't sound realistic or even possible in acceptable time with  
> usual 'Internet' bandwidth).
> 
> But, usually it's acceptable (or even requirement) to display/return  
> only highlighting snippets for 30-50 results matches and retrieve whole  
> document (json source) only for a single document (when requested by  
> exact ID).
> 
> To be able to provide highlighting snippets for large textual  
> ('content') field, it needs to be stored.
> 
> Now we are in situation where because textual field is too big (makes  
> impassible to return json source for 30-50 results to a client) we need  
> to store it twice in index (once as a part of original json source in  
> \_source and second time as a 'content' field "store" : "yes" for  
> highlighting). This make index a lot bigger.
> 
> Also, if there is a requirement to display highlighting for all fields  
> (separate highlight snippets for each field where match occurred,  
> without mixing fields snippets -\> stored \_all field can not be used for  
> highlighting in such case) then whole document (all fields) needs to be  
> stored twice.
> 
> In this case seems only logical to disable \_source field (since all  
> fields are stored anyway) and when whole document needs to be retrieved  
> use (newly added) fields=\* feature (I've read similar discussion thread  
> which led to this enhancement)
> 
> Here is my question/proposal:
> 
> would it be possible to enable use of json \_source field for 'field  
> specific' highlighting, where matching snippet needs to be returned  
> separately for each field name?
> 
> Maybe to have term\_vector for each field, but to somehow 'adjust' or  
> recalculate positions\_offsets to point to text snippet in \_source  
> instead of stored field?
> 
> I know this is not a simple requirement, but if 'field specific'  
> highlighting could somehow use stored json instead of requiring an  
> individual field to be (separately) stored, that would make a great  
> use/reuse of stored json \_source (and no one would ever think of  
> disabling it 🙂
> 
> Tomislav

---

<div class="post-metadata">

**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\
**Post date:** [August 12, 2010, 6:03pm UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213/3 "2010-08-12T18:03:33Z")

</div>

If I read it correctly then I think it partly overlaps with

> <https://github.com/elastic/elasticsearch/issues/308>
>
> I would like to see an option to get \`fields\` or \`\_source\` value with specific f…ields highlighted.
> One way how to get similar result now is using term query on an ID field (to get one specific document) which would allow to use highlight DSL but still getting full content of some field highlighted would require either NullFragmenter (which is not implemented now) or setting value of \`fragment\_size\` to some arbitrary large number.

[http://github.com/elasticsearch/elasticsearch/issues/issue/308](http://github.com/elasticsearch/elasticsearch/issues/issue/308)Lukas

On Thu, Aug 12, 2010 at 5:53 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> Agreed. A bit tricky to implement, but possible. Also, note that this will  
> require loading the full json from the index, and parse it in order to get  
> the relevant parts from it. It won't be returned, but still loaded. So, it  
> might not make sense when you have several big fields in the index, and you  
> want to get fragments for one of them. But does make sense when having one  
> big field.
> 
> Also, if this is implemented, it should also be possible to get specific  
> fields out of the json as a response as well (similar to asking for specific  
> fields in the search request, maybe call them source\_fileds).
> 
> Open an issue for this?
> 
> -shay.banon
> 
> On Thu, Aug 12, 2010 at 6:06 PM, Tomislav Poljak [tpoljak@gmail.com](mailto:tpoljak@gmail.com)wrote:
> 
> > Hi,  
> > I really like all the features stored json (enabled \_source) provides in  
> > both Java API (used it in indexing/updating) and REST API (used it for  
> > searching).
> > 
> > I do however have one possible request for improvement regarding  
> > documents with large textual fields and overall highlighting.
> > 
> > When there is a requirement to index/search documents with large textual  
> > fields (like 'content' with text in Mb, which is not unusual), returning  
> > a whole json for each result in result set can be impossible (if each  
> > json document has a few Mb in 'content' returning 30-50 results to a  
> > client doesn't sound realistic or even possible in acceptable time with  
> > usual 'Internet' bandwidth).
> > 
> > But, usually it's acceptable (or even requirement) to display/return  
> > only highlighting snippets for 30-50 results matches and retrieve whole  
> > document (json source) only for a single document (when requested by  
> > exact ID).
> > 
> > To be able to provide highlighting snippets for large textual  
> > ('content') field, it needs to be stored.
> > 
> > Now we are in situation where because textual field is too big (makes  
> > impassible to return json source for 30-50 results to a client) we need  
> > to store it twice in index (once as a part of original json source in  
> > \_source and second time as a 'content' field "store" : "yes" for  
> > highlighting). This make index a lot bigger.
> > 
> > Also, if there is a requirement to display highlighting for all fields  
> > (separate highlight snippets for each field where match occurred,  
> > without mixing fields snippets -\> stored \_all field can not be used for  
> > highlighting in such case) then whole document (all fields) needs to be  
> > stored twice.
> > 
> > In this case seems only logical to disable \_source field (since all  
> > fields are stored anyway) and when whole document needs to be retrieved  
> > use (newly added) fields=\* feature (I've read similar discussion thread  
> > which led to this enhancement)
> > 
> > Here is my question/proposal:
> > 
> > would it be possible to enable use of json \_source field for 'field  
> > specific' highlighting, where matching snippet needs to be returned  
> > separately for each field name?
> > 
> > Maybe to have term\_vector for each field, but to somehow 'adjust' or  
> > recalculate positions\_offsets to point to text snippet in \_source  
> > instead of stored field?
> > 
> > I know this is not a simple requirement, but if 'field specific'  
> > highlighting could somehow use stored json instead of requiring an  
> > individual field to be (separately) stored, that would make a great  
> > use/reuse of stored json \_source (and no one would ever think of  
> > disabling it 🙂
> > 
> > Tomislav

---

<div class="post-metadata">

**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\
**Post date:** [August 12, 2010, 6:08pm UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213/4 "2010-08-12T18:08:57Z")

</div>

One of differences is that the 308 issue was meant to return whole content  
of the \_source or some of its fields (or stored fields if not using  
"\_source"). But the point is that the user should be able to specify  
Fragmenter type (or provide custom implementation of Fragmenter).

On Thu, Aug 12, 2010 at 8:03 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote:

> If I read it correctly then I think it partly overlaps with  
> [Highlighting API: Apply highlighting to REST GET operation · Issue #308 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/308)
> 
> [http://github.com/elasticsearch/elasticsearch/issues/issue/308](http://github.com/elasticsearch/elasticsearch/issues/issue/308)Lukas
> 
> On Thu, Aug 12, 2010 at 5:53 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:
> 
> > Agreed. A bit tricky to implement, but possible. Also, note that this will  
> > require loading the full json from the index, and parse it in order to get  
> > the relevant parts from it. It won't be returned, but still loaded. So, it  
> > might not make sense when you have several big fields in the index, and you  
> > want to get fragments for one of them. But does make sense when having one  
> > big field.
> > 
> > Also, if this is implemented, it should also be possible to get specific  
> > fields out of the json as a response as well (similar to asking for specific  
> > fields in the search request, maybe call them source\_fileds).
> > 
> > Open an issue for this?
> > 
> > -shay.banon
> > 
> > On Thu, Aug 12, 2010 at 6:06 PM, Tomislav Poljak [tpoljak@gmail.com](mailto:tpoljak@gmail.com)wrote:
> > 
> > > Hi,  
> > > I really like all the features stored json (enabled \_source) provides in  
> > > both Java API (used it in indexing/updating) and REST API (used it for  
> > > searching).
> > > 
> > > I do however have one possible request for improvement regarding  
> > > documents with large textual fields and overall highlighting.
> > > 
> > > When there is a requirement to index/search documents with large textual  
> > > fields (like 'content' with text in Mb, which is not unusual), returning  
> > > a whole json for each result in result set can be impossible (if each  
> > > json document has a few Mb in 'content' returning 30-50 results to a  
> > > client doesn't sound realistic or even possible in acceptable time with  
> > > usual 'Internet' bandwidth).
> > > 
> > > But, usually it's acceptable (or even requirement) to display/return  
> > > only highlighting snippets for 30-50 results matches and retrieve whole  
> > > document (json source) only for a single document (when requested by  
> > > exact ID).
> > > 
> > > To be able to provide highlighting snippets for large textual  
> > > ('content') field, it needs to be stored.
> > > 
> > > Now we are in situation where because textual field is too big (makes  
> > > impassible to return json source for 30-50 results to a client) we need  
> > > to store it twice in index (once as a part of original json source in  
> > > \_source and second time as a 'content' field "store" : "yes" for  
> > > highlighting). This make index a lot bigger.
> > > 
> > > Also, if there is a requirement to display highlighting for all fields  
> > > (separate highlight snippets for each field where match occurred,  
> > > without mixing fields snippets -\> stored \_all field can not be used for  
> > > highlighting in such case) then whole document (all fields) needs to be  
> > > stored twice.
> > > 
> > > In this case seems only logical to disable \_source field (since all  
> > > fields are stored anyway) and when whole document needs to be retrieved  
> > > use (newly added) fields=\* feature (I've read similar discussion thread  
> > > which led to this enhancement)
> > > 
> > > Here is my question/proposal:
> > > 
> > > would it be possible to enable use of json \_source field for 'field  
> > > specific' highlighting, where matching snippet needs to be returned  
> > > separately for each field name?
> > > 
> > > Maybe to have term\_vector for each field, but to somehow 'adjust' or  
> > > recalculate positions\_offsets to point to text snippet in \_source  
> > > instead of stored field?
> > > 
> > > I know this is not a simple requirement, but if 'field specific'  
> > > highlighting could somehow use stored json instead of requiring an  
> > > individual field to be (separately) stored, that would make a great  
> > > use/reuse of stored json \_source (and no one would ever think of  
> > > disabling it 🙂
> > > 
> > > Tomislav

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 12, 2010, 6:41pm UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213/5 "2010-08-12T18:41:55Z")

</div>

Not sure if it overlaps, fragmenter controls how to break the highlighted  
data, this relates to how to fetch that date to highlight.

-shay.banon

On Thu, Aug 12, 2010 at 9:08 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote:

> One of differences is that the 308 issue was meant to return whole content  
> of the \_source or some of its fields (or stored fields if not using  
> "\_source"). But the point is that the user should be able to specify  
> Fragmenter type (or provide custom implementation of Fragmenter).
> 
> On Thu, Aug 12, 2010 at 8:03 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> 
> > If I read it correctly then I think it partly overlaps with  
> > [Highlighting API: Apply highlighting to REST GET operation · Issue #308 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/308)
> > 
> > [http://github.com/elasticsearch/elasticsearch/issues/issue/308](http://github.com/elasticsearch/elasticsearch/issues/issue/308)Lukas
> > 
> > On Thu, Aug 12, 2010 at 5:53 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> > 
> > > wrote:
> > 
> > > Agreed. A bit tricky to implement, but possible. Also, note that this  
> > > will require loading the full json from the index, and parse it in order to  
> > > get the relevant parts from it. It won't be returned, but still loaded. So,  
> > > it might not make sense when you have several big fields in the index, and  
> > > you want to get fragments for one of them. But does make sense when having  
> > > one big field.
> > > 
> > > Also, if this is implemented, it should also be possible to get specific  
> > > fields out of the json as a response as well (similar to asking for specific  
> > > fields in the search request, maybe call them source\_fileds).
> > > 
> > > Open an issue for this?
> > > 
> > > -shay.banon
> > > 
> > > On Thu, Aug 12, 2010 at 6:06 PM, Tomislav Poljak [tpoljak@gmail.com](mailto:tpoljak@gmail.com)wrote:
> > > 
> > > > Hi,  
> > > > I really like all the features stored json (enabled \_source) provides in  
> > > > both Java API (used it in indexing/updating) and REST API (used it for  
> > > > searching).
> > > > 
> > > > I do however have one possible request for improvement regarding  
> > > > documents with large textual fields and overall highlighting.
> > > > 
> > > > When there is a requirement to index/search documents with large textual  
> > > > fields (like 'content' with text in Mb, which is not unusual), returning  
> > > > a whole json for each result in result set can be impossible (if each  
> > > > json document has a few Mb in 'content' returning 30-50 results to a  
> > > > client doesn't sound realistic or even possible in acceptable time with  
> > > > usual 'Internet' bandwidth).
> > > > 
> > > > But, usually it's acceptable (or even requirement) to display/return  
> > > > only highlighting snippets for 30-50 results matches and retrieve whole  
> > > > document (json source) only for a single document (when requested by  
> > > > exact ID).
> > > > 
> > > > To be able to provide highlighting snippets for large textual  
> > > > ('content') field, it needs to be stored.
> > > > 
> > > > Now we are in situation where because textual field is too big (makes  
> > > > impassible to return json source for 30-50 results to a client) we need  
> > > > to store it twice in index (once as a part of original json source in  
> > > > \_source and second time as a 'content' field "store" : "yes" for  
> > > > highlighting). This make index a lot bigger.
> > > > 
> > > > Also, if there is a requirement to display highlighting for all fields  
> > > > (separate highlight snippets for each field where match occurred,  
> > > > without mixing fields snippets -\> stored \_all field can not be used for  
> > > > highlighting in such case) then whole document (all fields) needs to be  
> > > > stored twice.
> > > > 
> > > > In this case seems only logical to disable \_source field (since all  
> > > > fields are stored anyway) and when whole document needs to be retrieved  
> > > > use (newly added) fields=\* feature (I've read similar discussion thread  
> > > > which led to this enhancement)
> > > > 
> > > > Here is my question/proposal:
> > > > 
> > > > would it be possible to enable use of json \_source field for 'field  
> > > > specific' highlighting, where matching snippet needs to be returned  
> > > > separately for each field name?
> > > > 
> > > > Maybe to have term\_vector for each field, but to somehow 'adjust' or  
> > > > recalculate positions\_offsets to point to text snippet in \_source  
> > > > instead of stored field?
> > > > 
> > > > I know this is not a simple requirement, but if 'field specific'  
> > > > highlighting could somehow use stored json instead of requiring an  
> > > > individual field to be (separately) stored, that would make a great  
> > > > use/reuse of stored json \_source (and no one would ever think of  
> > > > disabling it 🙂
> > > > 
> > > > Tomislav

---

<div class="post-metadata">

**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\
**Post date:** [August 12, 2010, 6:53pm UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213/6 "2010-08-12T18:53:50Z")

</div>

Actually, that ticket has two parts. One is Fragmenter related and the other  
one is possibility to tell, that I want to highlight some portion of \_source  
data. Imagine I am using only REST API and for example if \_source is a  
person with name, address and bio fields then I would like to tell that I  
want to highlight just the bio field (and I think the NullFragmenter would  
be needed for this if I want to display whole content of bio highlighted,  
not just fragments). The other possibility would be to define mapping for  
person in such a way that bio would be a stored field, then I could query  
for stored fields (not pulling the \_source field) and tell the I want to  
apply NullFragmenter to this data while highlighting. But this gets back to  
the Tomislav's situation, because this would mean that bio is probably  
stored twice, once as a part of source and then separately as a stored bio  
field.

Lukas

On Thu, Aug 12, 2010 at 8:41 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> Not sure if it overlaps, fragmenter controls how to break the highlighted  
> data, this relates to how to fetch that date to highlight.
> 
> -shay.banon
> 
> On Thu, Aug 12, 2010 at 9:08 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> 
> > One of differences is that the 308 issue was meant to return whole content  
> > of the \_source or some of its fields (or stored fields if not using  
> > "\_source"). But the point is that the user should be able to specify  
> > Fragmenter type (or provide custom implementation of Fragmenter).
> > 
> > On Thu, Aug 12, 2010 at 8:03 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > 
> > > If I read it correctly then I think it partly overlaps with  
> > > [Highlighting API: Apply highlighting to REST GET operation · Issue #308 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/308)
> > > 
> > > [http://github.com/elasticsearch/elasticsearch/issues/issue/308](http://github.com/elasticsearch/elasticsearch/issues/issue/308)Lukas
> > > 
> > > On Thu, Aug 12, 2010 at 5:53 PM, Shay Banon \<  
> > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > 
> > > > Agreed. A bit tricky to implement, but possible. Also, note that this  
> > > > will require loading the full json from the index, and parse it in order to  
> > > > get the relevant parts from it. It won't be returned, but still loaded. So,  
> > > > it might not make sense when you have several big fields in the index, and  
> > > > you want to get fragments for one of them. But does make sense when having  
> > > > one big field.
> > > > 
> > > > Also, if this is implemented, it should also be possible to get specific  
> > > > fields out of the json as a response as well (similar to asking for specific  
> > > > fields in the search request, maybe call them source\_fileds).
> > > > 
> > > > Open an issue for this?
> > > > 
> > > > -shay.banon
> > > > 
> > > > On Thu, Aug 12, 2010 at 6:06 PM, Tomislav Poljak [tpoljak@gmail.com](mailto:tpoljak@gmail.com)wrote:
> > > > 
> > > > > Hi,  
> > > > > I really like all the features stored json (enabled \_source) provides  
> > > > > in  
> > > > > both Java API (used it in indexing/updating) and REST API (used it for  
> > > > > searching).
> > > > > 
> > > > > I do however have one possible request for improvement regarding  
> > > > > documents with large textual fields and overall highlighting.
> > > > > 
> > > > > When there is a requirement to index/search documents with large  
> > > > > textual  
> > > > > fields (like 'content' with text in Mb, which is not unusual),  
> > > > > returning  
> > > > > a whole json for each result in result set can be impossible (if each  
> > > > > json document has a few Mb in 'content' returning 30-50 results to a  
> > > > > client doesn't sound realistic or even possible in acceptable time with  
> > > > > usual 'Internet' bandwidth).
> > > > > 
> > > > > But, usually it's acceptable (or even requirement) to display/return  
> > > > > only highlighting snippets for 30-50 results matches and retrieve whole  
> > > > > document (json source) only for a single document (when requested by  
> > > > > exact ID).
> > > > > 
> > > > > To be able to provide highlighting snippets for large textual  
> > > > > ('content') field, it needs to be stored.
> > > > > 
> > > > > Now we are in situation where because textual field is too big (makes  
> > > > > impassible to return json source for 30-50 results to a client) we need  
> > > > > to store it twice in index (once as a part of original json source in  
> > > > > \_source and second time as a 'content' field "store" : "yes" for  
> > > > > highlighting). This make index a lot bigger.
> > > > > 
> > > > > Also, if there is a requirement to display highlighting for all fields  
> > > > > (separate highlight snippets for each field where match occurred,  
> > > > > without mixing fields snippets -\> stored \_all field can not be used for  
> > > > > highlighting in such case) then whole document (all fields) needs to be  
> > > > > stored twice.
> > > > > 
> > > > > In this case seems only logical to disable \_source field (since all  
> > > > > fields are stored anyway) and when whole document needs to be retrieved  
> > > > > use (newly added) fields=\* feature (I've read similar discussion thread  
> > > > > which led to this enhancement)
> > > > > 
> > > > > Here is my question/proposal:
> > > > > 
> > > > > would it be possible to enable use of json \_source field for 'field  
> > > > > specific' highlighting, where matching snippet needs to be returned  
> > > > > separately for each field name?
> > > > > 
> > > > > Maybe to have term\_vector for each field, but to somehow 'adjust' or  
> > > > > recalculate positions\_offsets to point to text snippet in \_source  
> > > > > instead of stored field?
> > > > > 
> > > > > I know this is not a simple requirement, but if 'field specific'  
> > > > > highlighting could somehow use stored json instead of requiring an  
> > > > > individual field to be (separately) stored, that would make a great  
> > > > > use/reuse of stored json \_source (and no one would ever think of  
> > > > > disabling it 🙂
> > > > > 
> > > > > Tomislav

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 12, 2010, 6:58pm UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213/7 "2010-08-12T18:58:57Z")

</div>

So, what you want is to be able to get just the bio field, without the full  
source, and without the bio field being stored? If so, then the response I  
gave, where the logic might apply also to get fields using something like  
"source\_field" notion applies here. It does mean that the full source will  
need to be retrieved and parsed. Not sure how highlighting comes into play  
here...

-shay.banon

On Thu, Aug 12, 2010 at 9:53 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote:

> Actually, that ticket has two parts. One is Fragmenter related and the  
> other one is possibility to tell, that I want to highlight some portion of  
> \_source data. Imagine I am using only REST API and for example if \_source is  
> a person with name, address and bio fields then I would like to tell that I  
> want to highlight just the bio field (and I think the NullFragmenter would  
> be needed for this if I want to display whole content of bio highlighted,  
> not just fragments). The other possibility would be to define mapping for  
> person in such a way that bio would be a stored field, then I could query  
> for stored fields (not pulling the \_source field) and tell the I want to  
> apply NullFragmenter to this data while highlighting. But this gets back to  
> the Tomislav's situation, because this would mean that bio is probably  
> stored twice, once as a part of source and then separately as a stored bio  
> field.
> 
> Lukas
> 
> On Thu, Aug 12, 2010 at 8:41 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:
> 
> > Not sure if it overlaps, fragmenter controls how to break the highlighted  
> > data, this relates to how to fetch that date to highlight.
> > 
> > -shay.banon
> > 
> > On Thu, Aug 12, 2010 at 9:08 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > 
> > > One of differences is that the 308 issue was meant to return whole  
> > > content of the \_source or some of its fields (or stored fields if not using  
> > > "\_source"). But the point is that the user should be able to specify  
> > > Fragmenter type (or provide custom implementation of Fragmenter).
> > > 
> > > On Thu, Aug 12, 2010 at 8:03 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > 
> > > > If I read it correctly then I think it partly overlaps with  
> > > > [Highlighting API: Apply highlighting to REST GET operation · Issue #308 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/308)
> > > > 
> > > > [http://github.com/elasticsearch/elasticsearch/issues/issue/308](http://github.com/elasticsearch/elasticsearch/issues/issue/308)Lukas
> > > > 
> > > > On Thu, Aug 12, 2010 at 5:53 PM, Shay Banon \<  
> > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > 
> > > > > Agreed. A bit tricky to implement, but possible. Also, note that this  
> > > > > will require loading the full json from the index, and parse it in order to  
> > > > > get the relevant parts from it. It won't be returned, but still loaded. So,  
> > > > > it might not make sense when you have several big fields in the index, and  
> > > > > you want to get fragments for one of them. But does make sense when having  
> > > > > one big field.
> > > > > 
> > > > > Also, if this is implemented, it should also be possible to get  
> > > > > specific fields out of the json as a response as well (similar to asking for  
> > > > > specific fields in the search request, maybe call them source\_fileds).
> > > > > 
> > > > > Open an issue for this?
> > > > > 
> > > > > -shay.banon
> > > > > 
> > > > > On Thu, Aug 12, 2010 at 6:06 PM, Tomislav Poljak [tpoljak@gmail.com](mailto:tpoljak@gmail.com)wrote:
> > > > > 
> > > > > > Hi,  
> > > > > > I really like all the features stored json (enabled \_source) provides  
> > > > > > in  
> > > > > > both Java API (used it in indexing/updating) and REST API (used it for  
> > > > > > searching).
> > > > > > 
> > > > > > I do however have one possible request for improvement regarding  
> > > > > > documents with large textual fields and overall highlighting.
> > > > > > 
> > > > > > When there is a requirement to index/search documents with large  
> > > > > > textual  
> > > > > > fields (like 'content' with text in Mb, which is not unusual),  
> > > > > > returning  
> > > > > > a whole json for each result in result set can be impossible (if each  
> > > > > > json document has a few Mb in 'content' returning 30-50 results to a  
> > > > > > client doesn't sound realistic or even possible in acceptable time  
> > > > > > with  
> > > > > > usual 'Internet' bandwidth).
> > > > > > 
> > > > > > But, usually it's acceptable (or even requirement) to display/return  
> > > > > > only highlighting snippets for 30-50 results matches and retrieve  
> > > > > > whole  
> > > > > > document (json source) only for a single document (when requested by  
> > > > > > exact ID).
> > > > > > 
> > > > > > To be able to provide highlighting snippets for large textual  
> > > > > > ('content') field, it needs to be stored.
> > > > > > 
> > > > > > Now we are in situation where because textual field is too big (makes  
> > > > > > impassible to return json source for 30-50 results to a client) we  
> > > > > > need  
> > > > > > to store it twice in index (once as a part of original json source in  
> > > > > > \_source and second time as a 'content' field "store" : "yes" for  
> > > > > > highlighting). This make index a lot bigger.
> > > > > > 
> > > > > > Also, if there is a requirement to display highlighting for all fields  
> > > > > > (separate highlight snippets for each field where match occurred,  
> > > > > > without mixing fields snippets -\> stored \_all field can not be used  
> > > > > > for  
> > > > > > highlighting in such case) then whole document (all fields) needs to  
> > > > > > be  
> > > > > > stored twice.
> > > > > > 
> > > > > > In this case seems only logical to disable \_source field (since all  
> > > > > > fields are stored anyway) and when whole document needs to be  
> > > > > > retrieved  
> > > > > > use (newly added) fields=\* feature (I've read similar discussion  
> > > > > > thread  
> > > > > > which led to this enhancement)
> > > > > > 
> > > > > > Here is my question/proposal:
> > > > > > 
> > > > > > would it be possible to enable use of json \_source field for 'field  
> > > > > > specific' highlighting, where matching snippet needs to be returned  
> > > > > > separately for each field name?
> > > > > > 
> > > > > > Maybe to have term\_vector for each field, but to somehow 'adjust' or  
> > > > > > recalculate positions\_offsets to point to text snippet in \_source  
> > > > > > instead of stored field?
> > > > > > 
> > > > > > I know this is not a simple requirement, but if 'field specific'  
> > > > > > highlighting could somehow use stored json instead of requiring an  
> > > > > > individual field to be (separately) stored, that would make a great  
> > > > > > use/reuse of stored json \_source (and no one would ever think of  
> > > > > > disabling it 🙂
> > > > > > 
> > > > > > Tomislav

---

<div class="post-metadata">

**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\
**Post date:** [August 12, 2010, 7:08pm UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213/8 "2010-08-12T19:08:14Z")

</div>

Imagine search app for HR: Candidate catalog (cool name!).  
The entities stored in the index are as follows: person: { id, name,  
address, bio }  
Now I am using just the REST API. Say I search for "Java" and I would like  
to display list of Names and allow users to click individual name which  
would display whole bio with Java highlighted in it (here comes the  
highlighting in the play!). Now I can display bio (just using GET REST API  
with given document ID but not highlighted. So I was thinking that it would  
be cool to have this function.

Lukas

2010/8/12 Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)

> So, what you want is to be able to get just the bio field, without the full  
> source, and without the bio field being stored? If so, then the response I  
> gave, where the logic might apply also to get fields using something like  
> "source\_field" notion applies here. It does mean that the full source will  
> need to be retrieved and parsed. Not sure how highlighting comes into play  
> here...
> 
> -shay.banon
> 
> On Thu, Aug 12, 2010 at 9:53 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> 
> > Actually, that ticket has two parts. One is Fragmenter related and the  
> > other one is possibility to tell, that I want to highlight some portion of  
> > \_source data. Imagine I am using only REST API and for example if \_source is  
> > a person with name, address and bio fields then I would like to tell that I  
> > want to highlight just the bio field (and I think the NullFragmenter would  
> > be needed for this if I want to display whole content of bio highlighted,  
> > not just fragments). The other possibility would be to define mapping for  
> > person in such a way that bio would be a stored field, then I could query  
> > for stored fields (not pulling the \_source field) and tell the I want to  
> > apply NullFragmenter to this data while highlighting. But this gets back to  
> > the Tomislav's situation, because this would mean that bio is probably  
> > stored twice, once as a part of source and then separately as a stored bio  
> > field.
> > 
> > Lukas
> > 
> > On Thu, Aug 12, 2010 at 8:41 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> > 
> > > wrote:
> > 
> > > Not sure if it overlaps, fragmenter controls how to break the highlighted  
> > > data, this relates to how to fetch that date to highlight.
> > > 
> > > -shay.banon
> > > 
> > > On Thu, Aug 12, 2010 at 9:08 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > 
> > > > One of differences is that the 308 issue was meant to return whole  
> > > > content of the \_source or some of its fields (or stored fields if not using  
> > > > "\_source"). But the point is that the user should be able to specify  
> > > > Fragmenter type (or provide custom implementation of Fragmenter).
> > > > 
> > > > On Thu, Aug 12, 2010 at 8:03 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > > 
> > > > > If I read it correctly then I think it partly overlaps with  
> > > > > [Highlighting API: Apply highlighting to REST GET operation · Issue #308 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/308)
> > > > > 
> > > > > [http://github.com/elasticsearch/elasticsearch/issues/issue/308](http://github.com/elasticsearch/elasticsearch/issues/issue/308)Lukas
> > > > > 
> > > > > On Thu, Aug 12, 2010 at 5:53 PM, Shay Banon \<  
> > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > 
> > > > > > Agreed. A bit tricky to implement, but possible. Also, note that this  
> > > > > > will require loading the full json from the index, and parse it in order to  
> > > > > > get the relevant parts from it. It won't be returned, but still loaded. So,  
> > > > > > it might not make sense when you have several big fields in the index, and  
> > > > > > you want to get fragments for one of them. But does make sense when having  
> > > > > > one big field.
> > > > > > 
> > > > > > Also, if this is implemented, it should also be possible to get  
> > > > > > specific fields out of the json as a response as well (similar to asking for  
> > > > > > specific fields in the search request, maybe call them source\_fileds).
> > > > > > 
> > > > > > Open an issue for this?
> > > > > > 
> > > > > > -shay.banon
> > > > > > 
> > > > > > On Thu, Aug 12, 2010 at 6:06 PM, Tomislav Poljak [tpoljak@gmail.com](mailto:tpoljak@gmail.com)wrote:
> > > > > > 
> > > > > > > Hi,  
> > > > > > > I really like all the features stored json (enabled \_source) provides  
> > > > > > > in  
> > > > > > > both Java API (used it in indexing/updating) and REST API (used it  
> > > > > > > for  
> > > > > > > searching).
> > > > > > > 
> > > > > > > I do however have one possible request for improvement regarding  
> > > > > > > documents with large textual fields and overall highlighting.
> > > > > > > 
> > > > > > > When there is a requirement to index/search documents with large  
> > > > > > > textual  
> > > > > > > fields (like 'content' with text in Mb, which is not unusual),  
> > > > > > > returning  
> > > > > > > a whole json for each result in result set can be impossible (if each  
> > > > > > > json document has a few Mb in 'content' returning 30-50 results to a  
> > > > > > > client doesn't sound realistic or even possible in acceptable time  
> > > > > > > with  
> > > > > > > usual 'Internet' bandwidth).
> > > > > > > 
> > > > > > > But, usually it's acceptable (or even requirement) to display/return  
> > > > > > > only highlighting snippets for 30-50 results matches and retrieve  
> > > > > > > whole  
> > > > > > > document (json source) only for a single document (when requested by  
> > > > > > > exact ID).
> > > > > > > 
> > > > > > > To be able to provide highlighting snippets for large textual  
> > > > > > > ('content') field, it needs to be stored.
> > > > > > > 
> > > > > > > Now we are in situation where because textual field is too big (makes  
> > > > > > > impassible to return json source for 30-50 results to a client) we  
> > > > > > > need  
> > > > > > > to store it twice in index (once as a part of original json source in  
> > > > > > > \_source and second time as a 'content' field "store" : "yes" for  
> > > > > > > highlighting). This make index a lot bigger.
> > > > > > > 
> > > > > > > Also, if there is a requirement to display highlighting for all  
> > > > > > > fields  
> > > > > > > (separate highlight snippets for each field where match occurred,  
> > > > > > > without mixing fields snippets -\> stored \_all field can not be used  
> > > > > > > for  
> > > > > > > highlighting in such case) then whole document (all fields) needs to  
> > > > > > > be  
> > > > > > > stored twice.
> > > > > > > 
> > > > > > > In this case seems only logical to disable \_source field (since all  
> > > > > > > fields are stored anyway) and when whole document needs to be  
> > > > > > > retrieved  
> > > > > > > use (newly added) fields=\* feature (I've read similar discussion  
> > > > > > > thread  
> > > > > > > which led to this enhancement)
> > > > > > > 
> > > > > > > Here is my question/proposal:
> > > > > > > 
> > > > > > > would it be possible to enable use of json \_source field for 'field  
> > > > > > > specific' highlighting, where matching snippet needs to be returned  
> > > > > > > separately for each field name?
> > > > > > > 
> > > > > > > Maybe to have term\_vector for each field, but to somehow 'adjust' or  
> > > > > > > recalculate positions\_offsets to point to text snippet in \_source  
> > > > > > > instead of stored field?
> > > > > > > 
> > > > > > > I know this is not a simple requirement, but if 'field specific'  
> > > > > > > highlighting could somehow use stored json instead of requiring an  
> > > > > > > individual field to be (separately) stored, that would make a great  
> > > > > > > use/reuse of stored json \_source (and no one would ever think of  
> > > > > > > disabling it 🙂
> > > > > > > 
> > > > > > > Tomislav

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 12, 2010, 7:14pm UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213/9 "2010-08-12T19:14:02Z")

</div>

Ahh, I see. So you would still need to provide a query to the GET api in  
order to do the highlighting, right?

On Thu, Aug 12, 2010 at 10:08 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote:

> Imagine search app for HR: Candidate catalog (cool name!).  
> The entities stored in the index are as follows: person: { id, name,  
> address, bio }  
> Now I am using just the REST API. Say I search for "Java" and I would like  
> to display list of Names and allow users to click individual name which  
> would display whole bio with Java highlighted in it (here comes the  
> highlighting in the play!). Now I can display bio (just using GET REST API  
> with given document ID but not highlighted. So I was thinking that it would  
> be cool to have this function.
> 
> Lukas
> 
> 2010/8/12 Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> 
> So, what you want is to be able to get just the bio field, without the full
> 
> > source, and without the bio field being stored? If so, then the response I  
> > gave, where the logic might apply also to get fields using something like  
> > "source\_field" notion applies here. It does mean that the full source will  
> > need to be retrieved and parsed. Not sure how highlighting comes into play  
> > here...
> > 
> > -shay.banon
> > 
> > On Thu, Aug 12, 2010 at 9:53 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > 
> > > Actually, that ticket has two parts. One is Fragmenter related and the  
> > > other one is possibility to tell, that I want to highlight some portion of  
> > > \_source data. Imagine I am using only REST API and for example if \_source is  
> > > a person with name, address and bio fields then I would like to tell that I  
> > > want to highlight just the bio field (and I think the NullFragmenter would  
> > > be needed for this if I want to display whole content of bio highlighted,  
> > > not just fragments). The other possibility would be to define mapping for  
> > > person in such a way that bio would be a stored field, then I could query  
> > > for stored fields (not pulling the \_source field) and tell the I want to  
> > > apply NullFragmenter to this data while highlighting. But this gets back to  
> > > the Tomislav's situation, because this would mean that bio is probably  
> > > stored twice, once as a part of source and then separately as a stored bio  
> > > field.
> > > 
> > > Lukas
> > > 
> > > On Thu, Aug 12, 2010 at 8:41 PM, Shay Banon \<  
> > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > 
> > > > Not sure if it overlaps, fragmenter controls how to break the  
> > > > highlighted data, this relates to how to fetch that date to highlight.
> > > > 
> > > > -shay.banon
> > > > 
> > > > On Thu, Aug 12, 2010 at 9:08 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > > 
> > > > > One of differences is that the 308 issue was meant to return whole  
> > > > > content of the \_source or some of its fields (or stored fields if not using  
> > > > > "\_source"). But the point is that the user should be able to specify  
> > > > > Fragmenter type (or provide custom implementation of Fragmenter).
> > > > > 
> > > > > On Thu, Aug 12, 2010 at 8:03 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > > > 
> > > > > > If I read it correctly then I think it partly overlaps with  
> > > > > > [Highlighting API: Apply highlighting to REST GET operation · Issue #308 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/308)
> > > > > > 
> > > > > > [http://github.com/elasticsearch/elasticsearch/issues/issue/308](http://github.com/elasticsearch/elasticsearch/issues/issue/308)Lukas
> > > > > > 
> > > > > > On Thu, Aug 12, 2010 at 5:53 PM, Shay Banon \<  
> > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > 
> > > > > > > Agreed. A bit tricky to implement, but possible. Also, note that this  
> > > > > > > will require loading the full json from the index, and parse it in order to  
> > > > > > > get the relevant parts from it. It won't be returned, but still loaded. So,  
> > > > > > > it might not make sense when you have several big fields in the index, and  
> > > > > > > you want to get fragments for one of them. But does make sense when having  
> > > > > > > one big field.
> > > > > > > 
> > > > > > > Also, if this is implemented, it should also be possible to get  
> > > > > > > specific fields out of the json as a response as well (similar to asking for  
> > > > > > > specific fields in the search request, maybe call them source\_fileds).
> > > > > > > 
> > > > > > > Open an issue for this?
> > > > > > > 
> > > > > > > -shay.banon
> > > > > > > 
> > > > > > > On Thu, Aug 12, 2010 at 6:06 PM, Tomislav Poljak [tpoljak@gmail.com](mailto:tpoljak@gmail.com)wrote:
> > > > > > > 
> > > > > > > > Hi,  
> > > > > > > > I really like all the features stored json (enabled \_source)  
> > > > > > > > provides in  
> > > > > > > > both Java API (used it in indexing/updating) and REST API (used it  
> > > > > > > > for  
> > > > > > > > searching).
> > > > > > > > 
> > > > > > > > I do however have one possible request for improvement regarding  
> > > > > > > > documents with large textual fields and overall highlighting.
> > > > > > > > 
> > > > > > > > When there is a requirement to index/search documents with large  
> > > > > > > > textual  
> > > > > > > > fields (like 'content' with text in Mb, which is not unusual),  
> > > > > > > > returning  
> > > > > > > > a whole json for each result in result set can be impossible (if  
> > > > > > > > each  
> > > > > > > > json document has a few Mb in 'content' returning 30-50 results to a  
> > > > > > > > client doesn't sound realistic or even possible in acceptable time  
> > > > > > > > with  
> > > > > > > > usual 'Internet' bandwidth).
> > > > > > > > 
> > > > > > > > But, usually it's acceptable (or even requirement) to display/return  
> > > > > > > > only highlighting snippets for 30-50 results matches and retrieve  
> > > > > > > > whole  
> > > > > > > > document (json source) only for a single document (when requested by  
> > > > > > > > exact ID).
> > > > > > > > 
> > > > > > > > To be able to provide highlighting snippets for large textual  
> > > > > > > > ('content') field, it needs to be stored.
> > > > > > > > 
> > > > > > > > Now we are in situation where because textual field is too big  
> > > > > > > > (makes  
> > > > > > > > impassible to return json source for 30-50 results to a client) we  
> > > > > > > > need  
> > > > > > > > to store it twice in index (once as a part of original json source  
> > > > > > > > in  
> > > > > > > > \_source and second time as a 'content' field "store" : "yes" for  
> > > > > > > > highlighting). This make index a lot bigger.
> > > > > > > > 
> > > > > > > > Also, if there is a requirement to display highlighting for all  
> > > > > > > > fields  
> > > > > > > > (separate highlight snippets for each field where match occurred,  
> > > > > > > > without mixing fields snippets -\> stored \_all field can not be used  
> > > > > > > > for  
> > > > > > > > highlighting in such case) then whole document (all fields) needs to  
> > > > > > > > be  
> > > > > > > > stored twice.
> > > > > > > > 
> > > > > > > > In this case seems only logical to disable \_source field (since all  
> > > > > > > > fields are stored anyway) and when whole document needs to be  
> > > > > > > > retrieved  
> > > > > > > > use (newly added) fields=\* feature (I've read similar discussion  
> > > > > > > > thread  
> > > > > > > > which led to this enhancement)
> > > > > > > > 
> > > > > > > > Here is my question/proposal:
> > > > > > > > 
> > > > > > > > would it be possible to enable use of json \_source field for 'field  
> > > > > > > > specific' highlighting, where matching snippet needs to be returned  
> > > > > > > > separately for each field name?
> > > > > > > > 
> > > > > > > > Maybe to have term\_vector for each field, but to somehow 'adjust' or  
> > > > > > > > recalculate positions\_offsets to point to text snippet in \_source  
> > > > > > > > instead of stored field?
> > > > > > > > 
> > > > > > > > I know this is not a simple requirement, but if 'field specific'  
> > > > > > > > highlighting could somehow use stored json instead of requiring an  
> > > > > > > > individual field to be (separately) stored, that would make a great  
> > > > > > > > use/reuse of stored json \_source (and no one would ever think of  
> > > > > > > > disabling it 🙂
> > > > > > > > 
> > > > > > > > Tomislav

---

<div class="post-metadata">

**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\
**Post date:** [August 12, 2010, 7:22pm UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213/10 "2010-08-12T19:22:04Z")

</div>

If I want to display whole bio highlighted then I can either get "\_source"  
and cut bio from it on the client side but in this case I need to tell ES to  
use highlighting on it first. Or I need to specify in mapping that bio is  
also stored and use fields query  
[http://www.elasticsearch.com/docs/elasticsearch/rest\_api/search/fields/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/search/fields/) but  
again I need to tell ES to highlight it. And in neither case I want only  
fragments, I want WHOLE content of the field. The first approach is not  
possible now the later is possible but required bio to be explicitly stored  
(and it is already stored in \_source).

Hope this makes it clear. (Sorry if I confused you).

Lukas

2010/8/12 Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)

> Ahh, I see. So you would still need to provide a query to the GET api in  
> order to do the highlighting, right?
> 
> On Thu, Aug 12, 2010 at 10:08 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> 
> > Imagine search app for HR: Candidate catalog (cool name!).  
> > The entities stored in the index are as follows: person: { id, name,  
> > address, bio }  
> > Now I am using just the REST API. Say I search for "Java" and I would like  
> > to display list of Names and allow users to click individual name which  
> > would display whole bio with Java highlighted in it (here comes the  
> > highlighting in the play!). Now I can display bio (just using GET REST API  
> > with given document ID but not highlighted. So I was thinking that it would  
> > be cool to have this function.
> > 
> > Lukas
> > 
> > 2010/8/12 Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> > 
> > So, what you want is to be able to get just the bio field, without the
> > 
> > > full source, and without the bio field being stored? If so, then the  
> > > response I gave, where the logic might apply also to get fields using  
> > > something like "source\_field" notion applies here. It does mean that the  
> > > full source will need to be retrieved and parsed. Not sure how highlighting  
> > > comes into play here...
> > > 
> > > -shay.banon
> > > 
> > > On Thu, Aug 12, 2010 at 9:53 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > 
> > > > Actually, that ticket has two parts. One is Fragmenter related and the  
> > > > other one is possibility to tell, that I want to highlight some portion of  
> > > > \_source data. Imagine I am using only REST API and for example if \_source is  
> > > > a person with name, address and bio fields then I would like to tell that I  
> > > > want to highlight just the bio field (and I think the NullFragmenter would  
> > > > be needed for this if I want to display whole content of bio highlighted,  
> > > > not just fragments). The other possibility would be to define mapping for  
> > > > person in such a way that bio would be a stored field, then I could query  
> > > > for stored fields (not pulling the \_source field) and tell the I want to  
> > > > apply NullFragmenter to this data while highlighting. But this gets back to  
> > > > the Tomislav's situation, because this would mean that bio is probably  
> > > > stored twice, once as a part of source and then separately as a stored bio  
> > > > field.
> > > > 
> > > > Lukas
> > > > 
> > > > On Thu, Aug 12, 2010 at 8:41 PM, Shay Banon \<  
> > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > 
> > > > > Not sure if it overlaps, fragmenter controls how to break the  
> > > > > highlighted data, this relates to how to fetch that date to highlight.
> > > > > 
> > > > > -shay.banon
> > > > > 
> > > > > On Thu, Aug 12, 2010 at 9:08 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > > > 
> > > > > > One of differences is that the 308 issue was meant to return whole  
> > > > > > content of the \_source or some of its fields (or stored fields if not using  
> > > > > > "\_source"). But the point is that the user should be able to specify  
> > > > > > Fragmenter type (or provide custom implementation of Fragmenter).
> > > > > > 
> > > > > > On Thu, Aug 12, 2010 at 8:03 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > > > > 
> > > > > > > If I read it correctly then I think it partly overlaps with  
> > > > > > > [Highlighting API: Apply highlighting to REST GET operation · Issue #308 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/308)
> > > > > > > 
> > > > > > > [http://github.com/elasticsearch/elasticsearch/issues/issue/308](http://github.com/elasticsearch/elasticsearch/issues/issue/308)  
> > > > > > > Lukas
> > > > > > > 
> > > > > > > On Thu, Aug 12, 2010 at 5:53 PM, Shay Banon \<  
> > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > > 
> > > > > > > > Agreed. A bit tricky to implement, but possible. Also, note that  
> > > > > > > > this will require loading the full json from the index, and parse it in  
> > > > > > > > order to get the relevant parts from it. It won't be returned, but still  
> > > > > > > > loaded. So, it might not make sense when you have several big fields in the  
> > > > > > > > index, and you want to get fragments for one of them. But does make sense  
> > > > > > > > when having one big field.
> > > > > > > > 
> > > > > > > > Also, if this is implemented, it should also be possible to get  
> > > > > > > > specific fields out of the json as a response as well (similar to asking for  
> > > > > > > > specific fields in the search request, maybe call them source\_fileds).
> > > > > > > > 
> > > > > > > > Open an issue for this?
> > > > > > > > 
> > > > > > > > -shay.banon
> > > > > > > > 
> > > > > > > > On Thu, Aug 12, 2010 at 6:06 PM, Tomislav Poljak \<[tpoljak@gmail.com](mailto:tpoljak@gmail.com)
> > > > > > > > 
> > > > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > Hi,  
> > > > > > > > > I really like all the features stored json (enabled \_source)  
> > > > > > > > > provides in  
> > > > > > > > > both Java API (used it in indexing/updating) and REST API (used it  
> > > > > > > > > for  
> > > > > > > > > searching).
> > > > > > > > > 
> > > > > > > > > I do however have one possible request for improvement regarding  
> > > > > > > > > documents with large textual fields and overall highlighting.
> > > > > > > > > 
> > > > > > > > > When there is a requirement to index/search documents with large  
> > > > > > > > > textual  
> > > > > > > > > fields (like 'content' with text in Mb, which is not unusual),  
> > > > > > > > > returning  
> > > > > > > > > a whole json for each result in result set can be impossible (if  
> > > > > > > > > each  
> > > > > > > > > json document has a few Mb in 'content' returning 30-50 results to  
> > > > > > > > > a  
> > > > > > > > > client doesn't sound realistic or even possible in acceptable time  
> > > > > > > > > with  
> > > > > > > > > usual 'Internet' bandwidth).
> > > > > > > > > 
> > > > > > > > > But, usually it's acceptable (or even requirement) to  
> > > > > > > > > display/return  
> > > > > > > > > only highlighting snippets for 30-50 results matches and retrieve  
> > > > > > > > > whole  
> > > > > > > > > document (json source) only for a single document (when requested  
> > > > > > > > > by  
> > > > > > > > > exact ID).
> > > > > > > > > 
> > > > > > > > > To be able to provide highlighting snippets for large textual  
> > > > > > > > > ('content') field, it needs to be stored.
> > > > > > > > > 
> > > > > > > > > Now we are in situation where because textual field is too big  
> > > > > > > > > (makes  
> > > > > > > > > impassible to return json source for 30-50 results to a client) we  
> > > > > > > > > need  
> > > > > > > > > to store it twice in index (once as a part of original json source  
> > > > > > > > > in  
> > > > > > > > > \_source and second time as a 'content' field "store" : "yes" for  
> > > > > > > > > highlighting). This make index a lot bigger.
> > > > > > > > > 
> > > > > > > > > Also, if there is a requirement to display highlighting for all  
> > > > > > > > > fields  
> > > > > > > > > (separate highlight snippets for each field where match occurred,  
> > > > > > > > > without mixing fields snippets -\> stored \_all field can not be used  
> > > > > > > > > for  
> > > > > > > > > highlighting in such case) then whole document (all fields) needs  
> > > > > > > > > to be  
> > > > > > > > > stored twice.
> > > > > > > > > 
> > > > > > > > > In this case seems only logical to disable \_source field (since all  
> > > > > > > > > fields are stored anyway) and when whole document needs to be  
> > > > > > > > > retrieved  
> > > > > > > > > use (newly added) fields=\* feature (I've read similar discussion  
> > > > > > > > > thread  
> > > > > > > > > which led to this enhancement)
> > > > > > > > > 
> > > > > > > > > Here is my question/proposal:
> > > > > > > > > 
> > > > > > > > > would it be possible to enable use of json \_source field for 'field  
> > > > > > > > > specific' highlighting, where matching snippet needs to be returned  
> > > > > > > > > separately for each field name?
> > > > > > > > > 
> > > > > > > > > Maybe to have term\_vector for each field, but to somehow 'adjust'  
> > > > > > > > > or  
> > > > > > > > > recalculate positions\_offsets to point to text snippet in \_source  
> > > > > > > > > instead of stored field?
> > > > > > > > > 
> > > > > > > > > I know this is not a simple requirement, but if 'field specific'  
> > > > > > > > > highlighting could somehow use stored json instead of requiring an  
> > > > > > > > > individual field to be (separately) stored, that would make a great  
> > > > > > > > > use/reuse of stored json \_source (and no one would ever think of  
> > > > > > > > > disabling it 🙂
> > > > > > > > > 
> > > > > > > > > Tomislav

---

<div class="post-metadata">

**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\
**Post date:** [August 12, 2010, 7:34pm UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213/11 "2010-08-12T19:34:45Z")

</div>

Oh, and one more note, see below:

On Thu, Aug 12, 2010 at 9:22 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote:

> If I want to display whole bio highlighted then I can either get "\_source"  
> and cut bio from it on the client side but in this case I need to tell ES to  
> use highlighting on it first. Or I need to specify in mapping that bio is  
> also stored and use fields query  
> [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/search/fields/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/search/fields/) but  
> again I need to tell ES to highlight it. And in neither case I want only  
> fragments, I want WHOLE content of the field. The first approach is not  
> possible now the later is possible but required bio to be explicitly stored  
> (and it is already stored in \_source).

And the later also requires specification of Fragmenter that returns whole  
body, not fragments, thus my reference to NullFragmenter, which is not  
implemented in FastVectorHighlighter API (as far as I understand it), it can  
be found in the older Highlighting API, thus I opened also

> <https://github.com/elastic/elasticsearch/issues/307>
>
> I would like to see possibility to use NullFragmenter and provide custom impleme…ntation of Fragmenter (a built-in Fragmenter that is aware of sentence boundaries would be cool).

May be it would be better if the NullFragmenter-like functionality is  
contributed directly into Lucene FastVectorHighlighter API. I was looking at  
the FVH API today and I think I can try to implement such Fragmenter.

> Hope this makes it clear. (Sorry if I confused you).
> 
> Lukas
> 
> 2010/8/12 Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> 
> > Ahh, I see. So you would still need to provide a query to the GET api in  
> > order to do the highlighting, right?
> > 
> > On Thu, Aug 12, 2010 at 10:08 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > 
> > > Imagine search app for HR: Candidate catalog (cool name!).  
> > > The entities stored in the index are as follows: person: { id, name,  
> > > address, bio }  
> > > Now I am using just the REST API. Say I search for "Java" and I would  
> > > like to display list of Names and allow users to click individual name which  
> > > would display whole bio with Java highlighted in it (here comes the  
> > > highlighting in the play!). Now I can display bio (just using GET REST API  
> > > with given document ID but not highlighted. So I was thinking that it would  
> > > be cool to have this function.
> > > 
> > > Lukas
> > > 
> > > 2010/8/12 Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> > > 
> > > So, what you want is to be able to get just the bio field, without the
> > > 
> > > > full source, and without the bio field being stored? If so, then the  
> > > > response I gave, where the logic might apply also to get fields using  
> > > > something like "source\_field" notion applies here. It does mean that the  
> > > > full source will need to be retrieved and parsed. Not sure how highlighting  
> > > > comes into play here...
> > > > 
> > > > -shay.banon
> > > > 
> > > > On Thu, Aug 12, 2010 at 9:53 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > > 
> > > > > Actually, that ticket has two parts. One is Fragmenter related and the  
> > > > > other one is possibility to tell, that I want to highlight some portion of  
> > > > > \_source data. Imagine I am using only REST API and for example if \_source is  
> > > > > a person with name, address and bio fields then I would like to tell that I  
> > > > > want to highlight just the bio field (and I think the NullFragmenter would  
> > > > > be needed for this if I want to display whole content of bio highlighted,  
> > > > > not just fragments). The other possibility would be to define mapping for  
> > > > > person in such a way that bio would be a stored field, then I could query  
> > > > > for stored fields (not pulling the \_source field) and tell the I want to  
> > > > > apply NullFragmenter to this data while highlighting. But this gets back to  
> > > > > the Tomislav's situation, because this would mean that bio is probably  
> > > > > stored twice, once as a part of source and then separately as a stored bio  
> > > > > field.
> > > > > 
> > > > > Lukas
> > > > > 
> > > > > On Thu, Aug 12, 2010 at 8:41 PM, Shay Banon \<  
> > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > 
> > > > > > Not sure if it overlaps, fragmenter controls how to break the  
> > > > > > highlighted data, this relates to how to fetch that date to highlight.
> > > > > > 
> > > > > > -shay.banon
> > > > > > 
> > > > > > On Thu, Aug 12, 2010 at 9:08 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > > > > 
> > > > > > > One of differences is that the 308 issue was meant to return whole  
> > > > > > > content of the \_source or some of its fields (or stored fields if not using  
> > > > > > > "\_source"). But the point is that the user should be able to specify  
> > > > > > > Fragmenter type (or provide custom implementation of Fragmenter).
> > > > > > > 
> > > > > > > On Thu, Aug 12, 2010 at 8:03 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > > > > > 
> > > > > > > > If I read it correctly then I think it partly overlaps with  
> > > > > > > > [Highlighting API: Apply highlighting to REST GET operation · Issue #308 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/308)
> > > > > > > > 
> > > > > > > > [http://github.com/elasticsearch/elasticsearch/issues/issue/308](http://github.com/elasticsearch/elasticsearch/issues/issue/308)  
> > > > > > > > Lukas
> > > > > > > > 
> > > > > > > > On Thu, Aug 12, 2010 at 5:53 PM, Shay Banon \<  
> > > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > > > 
> > > > > > > > > Agreed. A bit tricky to implement, but possible. Also, note that  
> > > > > > > > > this will require loading the full json from the index, and parse it in  
> > > > > > > > > order to get the relevant parts from it. It won't be returned, but still  
> > > > > > > > > loaded. So, it might not make sense when you have several big fields in the  
> > > > > > > > > index, and you want to get fragments for one of them. But does make sense  
> > > > > > > > > when having one big field.
> > > > > > > > > 
> > > > > > > > > Also, if this is implemented, it should also be possible to get  
> > > > > > > > > specific fields out of the json as a response as well (similar to asking for  
> > > > > > > > > specific fields in the search request, maybe call them source\_fileds).
> > > > > > > > > 
> > > > > > > > > Open an issue for this?
> > > > > > > > > 
> > > > > > > > > -shay.banon
> > > > > > > > > 
> > > > > > > > > On Thu, Aug 12, 2010 at 6:06 PM, Tomislav Poljak \<  
> > > > > > > > > [tpoljak@gmail.com](mailto:tpoljak@gmail.com)\> wrote:
> > > > > > > > > 
> > > > > > > > > > Hi,  
> > > > > > > > > > I really like all the features stored json (enabled \_source)  
> > > > > > > > > > provides in  
> > > > > > > > > > both Java API (used it in indexing/updating) and REST API (used it  
> > > > > > > > > > for  
> > > > > > > > > > searching).
> > > > > > > > > > 
> > > > > > > > > > I do however have one possible request for improvement regarding  
> > > > > > > > > > documents with large textual fields and overall highlighting.
> > > > > > > > > > 
> > > > > > > > > > When there is a requirement to index/search documents with large  
> > > > > > > > > > textual  
> > > > > > > > > > fields (like 'content' with text in Mb, which is not unusual),  
> > > > > > > > > > returning  
> > > > > > > > > > a whole json for each result in result set can be impossible (if  
> > > > > > > > > > each  
> > > > > > > > > > json document has a few Mb in 'content' returning 30-50 results to  
> > > > > > > > > > a  
> > > > > > > > > > client doesn't sound realistic or even possible in acceptable time  
> > > > > > > > > > with  
> > > > > > > > > > usual 'Internet' bandwidth).
> > > > > > > > > > 
> > > > > > > > > > But, usually it's acceptable (or even requirement) to  
> > > > > > > > > > display/return  
> > > > > > > > > > only highlighting snippets for 30-50 results matches and retrieve  
> > > > > > > > > > whole  
> > > > > > > > > > document (json source) only for a single document (when requested  
> > > > > > > > > > by  
> > > > > > > > > > exact ID).
> > > > > > > > > > 
> > > > > > > > > > To be able to provide highlighting snippets for large textual  
> > > > > > > > > > ('content') field, it needs to be stored.
> > > > > > > > > > 
> > > > > > > > > > Now we are in situation where because textual field is too big  
> > > > > > > > > > (makes  
> > > > > > > > > > impassible to return json source for 30-50 results to a client) we  
> > > > > > > > > > need  
> > > > > > > > > > to store it twice in index (once as a part of original json source  
> > > > > > > > > > in  
> > > > > > > > > > \_source and second time as a 'content' field "store" : "yes" for  
> > > > > > > > > > highlighting). This make index a lot bigger.
> > > > > > > > > > 
> > > > > > > > > > Also, if there is a requirement to display highlighting for all  
> > > > > > > > > > fields  
> > > > > > > > > > (separate highlight snippets for each field where match occurred,  
> > > > > > > > > > without mixing fields snippets -\> stored \_all field can not be  
> > > > > > > > > > used for  
> > > > > > > > > > highlighting in such case) then whole document (all fields) needs  
> > > > > > > > > > to be  
> > > > > > > > > > stored twice.
> > > > > > > > > > 
> > > > > > > > > > In this case seems only logical to disable \_source field (since  
> > > > > > > > > > all  
> > > > > > > > > > fields are stored anyway) and when whole document needs to be  
> > > > > > > > > > retrieved  
> > > > > > > > > > use (newly added) fields=\* feature (I've read similar discussion  
> > > > > > > > > > thread  
> > > > > > > > > > which led to this enhancement)
> > > > > > > > > > 
> > > > > > > > > > Here is my question/proposal:
> > > > > > > > > > 
> > > > > > > > > > would it be possible to enable use of json \_source field for  
> > > > > > > > > > 'field  
> > > > > > > > > > specific' highlighting, where matching snippet needs to be  
> > > > > > > > > > returned  
> > > > > > > > > > separately for each field name?
> > > > > > > > > > 
> > > > > > > > > > Maybe to have term\_vector for each field, but to somehow 'adjust'  
> > > > > > > > > > or  
> > > > > > > > > > recalculate positions\_offsets to point to text snippet in \_source  
> > > > > > > > > > instead of stored field?
> > > > > > > > > > 
> > > > > > > > > > I know this is not a simple requirement, but if 'field specific'  
> > > > > > > > > > highlighting could somehow use stored json instead of requiring an  
> > > > > > > > > > individual field to be (separately) stored, that would make a  
> > > > > > > > > > great  
> > > > > > > > > > use/reuse of stored json \_source (and no one would ever think of  
> > > > > > > > > > disabling it 🙂
> > > > > > > > > > 
> > > > > > > > > > Tomislav

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 12, 2010, 10:30pm UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213/12 "2010-08-12T22:30:32Z")

</div>

ok, so you want to get the whole bio field highlighted, so you would need to  
pass the query to the get API as well, otherwise, there is no way to  
highlight it (aside from other things you need, like the option to do no  
fragmentation and getting the actual data).

On Thu, Aug 12, 2010 at 10:34 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote:

> Oh, and one more note, see below:
> 
> On Thu, Aug 12, 2010 at 9:22 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> 
> > If I want to display whole bio highlighted then I can either get "\_source"  
> > and cut bio from it on the client side but in this case I need to tell ES to  
> > use highlighting on it first. Or I need to specify in mapping that bio is  
> > also stored and use fields query  
> > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/search/fields/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/search/fields/) but  
> > again I need to tell ES to highlight it. And in neither case I want only  
> > fragments, I want WHOLE content of the field. The first approach is not  
> > possible now the later is possible but required bio to be explicitly stored  
> > (and it is already stored in \_source).
> 
> And the later also requires specification of Fragmenter that returns whole  
> body, not fragments, thus my reference to NullFragmenter, which is not  
> implemented in FastVectorHighlighter API (as far as I understand it), it can  
> be found in the older Highlighting API, thus I opened also  
> [Highlighting API: Add support for custom FragmentsBuilder · Issue #307 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/307)
> 
> May be it would be better if the NullFragmenter-like functionality is  
> contributed directly into Lucene FastVectorHighlighter API. I was looking at  
> the FVH API today and I think I can try to implement such Fragmenter.
> 
> > Hope this makes it clear. (Sorry if I confused you).
> > 
> > Lukas
> > 
> > 2010/8/12 Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> > 
> > > Ahh, I see. So you would still need to provide a query to the GET api in  
> > > order to do the highlighting, right?
> > > 
> > > On Thu, Aug 12, 2010 at 10:08 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > 
> > > > Imagine search app for HR: Candidate catalog (cool name!).  
> > > > The entities stored in the index are as follows: person: { id, name,  
> > > > address, bio }  
> > > > Now I am using just the REST API. Say I search for "Java" and I would  
> > > > like to display list of Names and allow users to click individual name which  
> > > > would display whole bio with Java highlighted in it (here comes the  
> > > > highlighting in the play!). Now I can display bio (just using GET REST API  
> > > > with given document ID but not highlighted. So I was thinking that it would  
> > > > be cool to have this function.
> > > > 
> > > > Lukas
> > > > 
> > > > 2010/8/12 Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> > > > 
> > > > So, what you want is to be able to get just the bio field, without the
> > > > 
> > > > > full source, and without the bio field being stored? If so, then the  
> > > > > response I gave, where the logic might apply also to get fields using  
> > > > > something like "source\_field" notion applies here. It does mean that the  
> > > > > full source will need to be retrieved and parsed. Not sure how highlighting  
> > > > > comes into play here...
> > > > > 
> > > > > -shay.banon
> > > > > 
> > > > > On Thu, Aug 12, 2010 at 9:53 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > > > 
> > > > > > Actually, that ticket has two parts. One is Fragmenter related and the  
> > > > > > other one is possibility to tell, that I want to highlight some portion of  
> > > > > > \_source data. Imagine I am using only REST API and for example if \_source is  
> > > > > > a person with name, address and bio fields then I would like to tell that I  
> > > > > > want to highlight just the bio field (and I think the NullFragmenter would  
> > > > > > be needed for this if I want to display whole content of bio highlighted,  
> > > > > > not just fragments). The other possibility would be to define mapping for  
> > > > > > person in such a way that bio would be a stored field, then I could query  
> > > > > > for stored fields (not pulling the \_source field) and tell the I want to  
> > > > > > apply NullFragmenter to this data while highlighting. But this gets back to  
> > > > > > the Tomislav's situation, because this would mean that bio is probably  
> > > > > > stored twice, once as a part of source and then separately as a stored bio  
> > > > > > field.
> > > > > > 
> > > > > > Lukas
> > > > > > 
> > > > > > On Thu, Aug 12, 2010 at 8:41 PM, Shay Banon \<  
> > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > 
> > > > > > > Not sure if it overlaps, fragmenter controls how to break the  
> > > > > > > highlighted data, this relates to how to fetch that date to highlight.
> > > > > > > 
> > > > > > > -shay.banon
> > > > > > > 
> > > > > > > On Thu, Aug 12, 2010 at 9:08 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > > > > > 
> > > > > > > > One of differences is that the 308 issue was meant to return whole  
> > > > > > > > content of the \_source or some of its fields (or stored fields if not using  
> > > > > > > > "\_source"). But the point is that the user should be able to specify  
> > > > > > > > Fragmenter type (or provide custom implementation of Fragmenter).
> > > > > > > > 
> > > > > > > > On Thu, Aug 12, 2010 at 8:03 PM, Lukáš Vlček \<[lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)
> > > > > > > > 
> > > > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > If I read it correctly then I think it partly overlaps with  
> > > > > > > > > [Highlighting API: Apply highlighting to REST GET operation · Issue #308 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/308)
> > > > > > > > > 
> > > > > > > > > [http://github.com/elasticsearch/elasticsearch/issues/issue/308](http://github.com/elasticsearch/elasticsearch/issues/issue/308)  
> > > > > > > > > Lukas
> > > > > > > > > 
> > > > > > > > > On Thu, Aug 12, 2010 at 5:53 PM, Shay Banon \<  
> > > > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > > > > 
> > > > > > > > > > Agreed. A bit tricky to implement, but possible. Also, note that  
> > > > > > > > > > this will require loading the full json from the index, and parse it in  
> > > > > > > > > > order to get the relevant parts from it. It won't be returned, but still  
> > > > > > > > > > loaded. So, it might not make sense when you have several big fields in the  
> > > > > > > > > > index, and you want to get fragments for one of them. But does make sense  
> > > > > > > > > > when having one big field.
> > > > > > > > > > 
> > > > > > > > > > Also, if this is implemented, it should also be possible to get  
> > > > > > > > > > specific fields out of the json as a response as well (similar to asking for  
> > > > > > > > > > specific fields in the search request, maybe call them source\_fileds).
> > > > > > > > > > 
> > > > > > > > > > Open an issue for this?
> > > > > > > > > > 
> > > > > > > > > > -shay.banon
> > > > > > > > > > 
> > > > > > > > > > On Thu, Aug 12, 2010 at 6:06 PM, Tomislav Poljak \<  
> > > > > > > > > > [tpoljak@gmail.com](mailto:tpoljak@gmail.com)\> wrote:
> > > > > > > > > > 
> > > > > > > > > > > Hi,  
> > > > > > > > > > > I really like all the features stored json (enabled \_source)  
> > > > > > > > > > > provides in  
> > > > > > > > > > > both Java API (used it in indexing/updating) and REST API (used  
> > > > > > > > > > > it for  
> > > > > > > > > > > searching).
> > > > > > > > > > > 
> > > > > > > > > > > I do however have one possible request for improvement regarding  
> > > > > > > > > > > documents with large textual fields and overall highlighting.
> > > > > > > > > > > 
> > > > > > > > > > > When there is a requirement to index/search documents with large  
> > > > > > > > > > > textual  
> > > > > > > > > > > fields (like 'content' with text in Mb, which is not unusual),  
> > > > > > > > > > > returning  
> > > > > > > > > > > a whole json for each result in result set can be impossible (if  
> > > > > > > > > > > each  
> > > > > > > > > > > json document has a few Mb in 'content' returning 30-50 results  
> > > > > > > > > > > to a  
> > > > > > > > > > > client doesn't sound realistic or even possible in acceptable  
> > > > > > > > > > > time with  
> > > > > > > > > > > usual 'Internet' bandwidth).
> > > > > > > > > > > 
> > > > > > > > > > > But, usually it's acceptable (or even requirement) to  
> > > > > > > > > > > display/return  
> > > > > > > > > > > only highlighting snippets for 30-50 results matches and retrieve  
> > > > > > > > > > > whole  
> > > > > > > > > > > document (json source) only for a single document (when requested  
> > > > > > > > > > > by  
> > > > > > > > > > > exact ID).
> > > > > > > > > > > 
> > > > > > > > > > > To be able to provide highlighting snippets for large textual  
> > > > > > > > > > > ('content') field, it needs to be stored.
> > > > > > > > > > > 
> > > > > > > > > > > Now we are in situation where because textual field is too big  
> > > > > > > > > > > (makes  
> > > > > > > > > > > impassible to return json source for 30-50 results to a client)  
> > > > > > > > > > > we need  
> > > > > > > > > > > to store it twice in index (once as a part of original json  
> > > > > > > > > > > source in  
> > > > > > > > > > > \_source and second time as a 'content' field "store" : "yes" for  
> > > > > > > > > > > highlighting). This make index a lot bigger.
> > > > > > > > > > > 
> > > > > > > > > > > Also, if there is a requirement to display highlighting for all  
> > > > > > > > > > > fields  
> > > > > > > > > > > (separate highlight snippets for each field where match occurred,  
> > > > > > > > > > > without mixing fields snippets -\> stored \_all field can not be  
> > > > > > > > > > > used for  
> > > > > > > > > > > highlighting in such case) then whole document (all fields) needs  
> > > > > > > > > > > to be  
> > > > > > > > > > > stored twice.
> > > > > > > > > > > 
> > > > > > > > > > > In this case seems only logical to disable \_source field (since  
> > > > > > > > > > > all  
> > > > > > > > > > > fields are stored anyway) and when whole document needs to be  
> > > > > > > > > > > retrieved  
> > > > > > > > > > > use (newly added) fields=\* feature (I've read similar discussion  
> > > > > > > > > > > thread  
> > > > > > > > > > > which led to this enhancement)
> > > > > > > > > > > 
> > > > > > > > > > > Here is my question/proposal:
> > > > > > > > > > > 
> > > > > > > > > > > would it be possible to enable use of json \_source field for  
> > > > > > > > > > > 'field  
> > > > > > > > > > > specific' highlighting, where matching snippet needs to be  
> > > > > > > > > > > returned  
> > > > > > > > > > > separately for each field name?
> > > > > > > > > > > 
> > > > > > > > > > > Maybe to have term\_vector for each field, but to somehow 'adjust'  
> > > > > > > > > > > or  
> > > > > > > > > > > recalculate positions\_offsets to point to text snippet in \_source  
> > > > > > > > > > > instead of stored field?
> > > > > > > > > > > 
> > > > > > > > > > > I know this is not a simple requirement, but if 'field specific'  
> > > > > > > > > > > highlighting could somehow use stored json instead of requiring  
> > > > > > > > > > > an  
> > > > > > > > > > > individual field to be (separately) stored, that would make a  
> > > > > > > > > > > great  
> > > > > > > > > > > use/reuse of stored json \_source (and no one would ever think of  
> > > > > > > > > > > disabling it 🙂
> > > > > > > > > > > 
> > > > > > > > > > > Tomislav

---

<div class="post-metadata">

**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\
**Post date:** [August 13, 2010, 8:20am UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213/13 "2010-08-13T08:20:00Z")

</div>

Yes, I did not realize that earlier but you are right that I will need to  
pass query into the highlight section as well.  
Take the following example:

I need to display all candidates that match "dude java" query and then I  
want to allow user to click on individual name and get whole bio  
highlighted.

So how I can go about this:  
First, I can get relevant documents using simple "query\_string" query for  
"dude java". I can now display names of candidates without highlights and  
highlighted fragments from bio for each name, kind of basic search interface  
that already works now. But if I wanted to display highlighted name I would  
get something like "..e _Dude_ Abid..." which is not what I want  
(sure, I can work with fragment size but that is just workaround and does  
not fit all situations). So when using that "query\_string" query I would  
like to specify in the highlight section that the person.name should be  
highlighted with no fragments.

Second, now, when the user clicks individual name, then I want to get whole  
bio highlighted.  
So I need to get specific document (by ID) and have the bio  
field highlighted (and the name field as well)  
The example of the query that could be used:

curl -XGET [http://localhost:9200/\_all/\_search](http://localhost:9200/_all/_search) -d '  
{ "query" : { "term" : { "person-id" : "1234" } },  
"highlight" : {  
"fields" : {  
"\_source" : {  
"path" : "person.bio,person.name",  
"fragmenter" : "classpath.to.NullFragmenter",  
"query" : {  
"query\_string" : { "fields" : ["bio","name"], "query" : "dude  
java" }  
}  
}  
}  
}  
}'

or I could use fields query:

curl -XGET [http://localhost:9200/\_all/\_search](http://localhost:9200/_all/_search) -d '  
{ "query" : { "term" : { "person-id" : "1234" } },  
"fields" : ["bio","name"],  
"highlight" : {  
"fields" : {  
"bio" : {  
"query" : {  
"query\_string" : { "fields" : ["bio"], "query" : "dude java" }  
}  
},  
"name" : {  
"query" : {  
"query\_string" : { "fields" : ["name"], "query" : "dude java" }  
}  
}  
}  
}  
}'

The later query requires both bio and name to be stored (and this is where  
it gets back to Tomislav's original point I think).  
Ugh! I am complicating it way too much... but hope the request is clear now  
🙂

Regards,  
Lukas

2010/8/13 Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)

> ok, so you want to get the whole bio field highlighted, so you would need  
> to pass the query to the get API as well, otherwise, there is no way to  
> highlight it (aside from other things you need, like the option to do no  
> fragmentation and getting the actual data).
> 
> On Thu, Aug 12, 2010 at 10:34 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> 
> > Oh, and one more note, see below:
> > 
> > On Thu, Aug 12, 2010 at 9:22 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > 
> > > If I want to display whole bio highlighted then I can either get  
> > > "\_source" and cut bio from it on the client side but in this case I need to  
> > > tell ES to use highlighting on it first. Or I need to specify in mapping  
> > > that bio is also stored and use fields query  
> > > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/search/fields/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/search/fields/) but  
> > > again I need to tell ES to highlight it. And in neither case I want only  
> > > fragments, I want WHOLE content of the field. The first approach is not  
> > > possible now the later is possible but required bio to be explicitly stored  
> > > (and it is already stored in \_source).
> > 
> > And the later also requires specification of Fragmenter that returns whole  
> > body, not fragments, thus my reference to NullFragmenter, which is not  
> > implemented in FastVectorHighlighter API (as far as I understand it), it can  
> > be found in the older Highlighting API, thus I opened also  
> > [Highlighting API: Add support for custom FragmentsBuilder · Issue #307 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/307)
> > 
> > May be it would be better if the NullFragmenter-like functionality is  
> > contributed directly into Lucene FastVectorHighlighter API. I was looking at  
> > the FVH API today and I think I can try to implement such Fragmenter.
> > 
> > > Hope this makes it clear. (Sorry if I confused you).
> > > 
> > > Lukas
> > > 
> > > 2010/8/12 Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> > > 
> > > > Ahh, I see. So you would still need to provide a query to the GET api in  
> > > > order to do the highlighting, right?
> > > > 
> > > > On Thu, Aug 12, 2010 at 10:08 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > > 
> > > > > Imagine search app for HR: Candidate catalog (cool name!).  
> > > > > The entities stored in the index are as follows: person: { id, name,  
> > > > > address, bio }  
> > > > > Now I am using just the REST API. Say I search for "Java" and I would  
> > > > > like to display list of Names and allow users to click individual name which  
> > > > > would display whole bio with Java highlighted in it (here comes the  
> > > > > highlighting in the play!). Now I can display bio (just using GET REST API  
> > > > > with given document ID but not highlighted. So I was thinking that it would  
> > > > > be cool to have this function.
> > > > > 
> > > > > Lukas
> > > > > 
> > > > > 2010/8/12 Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> > > > > 
> > > > > So, what you want is to be able to get just the bio field, without the
> > > > > 
> > > > > > full source, and without the bio field being stored? If so, then the  
> > > > > > response I gave, where the logic might apply also to get fields using  
> > > > > > something like "source\_field" notion applies here. It does mean that the  
> > > > > > full source will need to be retrieved and parsed. Not sure how highlighting  
> > > > > > comes into play here...
> > > > > > 
> > > > > > -shay.banon
> > > > > > 
> > > > > > On Thu, Aug 12, 2010 at 9:53 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > > > > 
> > > > > > > Actually, that ticket has two parts. One is Fragmenter related and  
> > > > > > > the other one is possibility to tell, that I want to highlight some portion  
> > > > > > > of \_source data. Imagine I am using only REST API and for example if \_source  
> > > > > > > is a person with name, address and bio fields then I would like to tell that  
> > > > > > > I want to highlight just the bio field (and I think the NullFragmenter would  
> > > > > > > be needed for this if I want to display whole content of bio highlighted,  
> > > > > > > not just fragments). The other possibility would be to define mapping for  
> > > > > > > person in such a way that bio would be a stored field, then I could query  
> > > > > > > for stored fields (not pulling the \_source field) and tell the I want to  
> > > > > > > apply NullFragmenter to this data while highlighting. But this gets back to  
> > > > > > > the Tomislav's situation, because this would mean that bio is probably  
> > > > > > > stored twice, once as a part of source and then separately as a stored bio  
> > > > > > > field.
> > > > > > > 
> > > > > > > Lukas
> > > > > > > 
> > > > > > > On Thu, Aug 12, 2010 at 8:41 PM, Shay Banon \<  
> > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > > 
> > > > > > > > Not sure if it overlaps, fragmenter controls how to break the  
> > > > > > > > highlighted data, this relates to how to fetch that date to highlight.
> > > > > > > > 
> > > > > > > > -shay.banon
> > > > > > > > 
> > > > > > > > On Thu, Aug 12, 2010 at 9:08 PM, Lukáš Vlček \<[lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)
> > > > > > > > 
> > > > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > One of differences is that the 308 issue was meant to return whole  
> > > > > > > > > content of the \_source or some of its fields (or stored fields if not using  
> > > > > > > > > "\_source"). But the point is that the user should be able to specify  
> > > > > > > > > Fragmenter type (or provide custom implementation of Fragmenter).
> > > > > > > > > 
> > > > > > > > > On Thu, Aug 12, 2010 at 8:03 PM, Lukáš Vlček \<  
> > > > > > > > > [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)\> wrote:
> > > > > > > > > 
> > > > > > > > > > If I read it correctly then I think it partly overlaps with  
> > > > > > > > > > [Highlighting API: Apply highlighting to REST GET operation · Issue #308 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/308)
> > > > > > > > > > 
> > > > > > > > > > [http://github.com/elasticsearch/elasticsearch/issues/issue/308](http://github.com/elasticsearch/elasticsearch/issues/issue/308)  
> > > > > > > > > > Lukas
> > > > > > > > > > 
> > > > > > > > > > On Thu, Aug 12, 2010 at 5:53 PM, Shay Banon \<  
> > > > > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > > > > > 
> > > > > > > > > > > Agreed. A bit tricky to implement, but possible. Also, note that  
> > > > > > > > > > > this will require loading the full json from the index, and parse it in  
> > > > > > > > > > > order to get the relevant parts from it. It won't be returned, but still  
> > > > > > > > > > > loaded. So, it might not make sense when you have several big fields in the  
> > > > > > > > > > > index, and you want to get fragments for one of them. But does make sense  
> > > > > > > > > > > when having one big field.
> > > > > > > > > > > 
> > > > > > > > > > > Also, if this is implemented, it should also be possible to get  
> > > > > > > > > > > specific fields out of the json as a response as well (similar to asking for  
> > > > > > > > > > > specific fields in the search request, maybe call them source\_fileds).
> > > > > > > > > > > 
> > > > > > > > > > > Open an issue for this?
> > > > > > > > > > > 
> > > > > > > > > > > -shay.banon
> > > > > > > > > > > 
> > > > > > > > > > > On Thu, Aug 12, 2010 at 6:06 PM, Tomislav Poljak \<  
> > > > > > > > > > > [tpoljak@gmail.com](mailto:tpoljak@gmail.com)\> wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > Hi,  
> > > > > > > > > > > > I really like all the features stored json (enabled \_source)  
> > > > > > > > > > > > provides in  
> > > > > > > > > > > > both Java API (used it in indexing/updating) and REST API (used  
> > > > > > > > > > > > it for  
> > > > > > > > > > > > searching).
> > > > > > > > > > > > 
> > > > > > > > > > > > I do however have one possible request for improvement regarding  
> > > > > > > > > > > > documents with large textual fields and overall highlighting.
> > > > > > > > > > > > 
> > > > > > > > > > > > When there is a requirement to index/search documents with large  
> > > > > > > > > > > > textual  
> > > > > > > > > > > > fields (like 'content' with text in Mb, which is not unusual),  
> > > > > > > > > > > > returning  
> > > > > > > > > > > > a whole json for each result in result set can be impossible (if  
> > > > > > > > > > > > each  
> > > > > > > > > > > > json document has a few Mb in 'content' returning 30-50 results  
> > > > > > > > > > > > to a  
> > > > > > > > > > > > client doesn't sound realistic or even possible in acceptable  
> > > > > > > > > > > > time with  
> > > > > > > > > > > > usual 'Internet' bandwidth).
> > > > > > > > > > > > 
> > > > > > > > > > > > But, usually it's acceptable (or even requirement) to  
> > > > > > > > > > > > display/return  
> > > > > > > > > > > > only highlighting snippets for 30-50 results matches and  
> > > > > > > > > > > > retrieve whole  
> > > > > > > > > > > > document (json source) only for a single document (when  
> > > > > > > > > > > > requested by  
> > > > > > > > > > > > exact ID).
> > > > > > > > > > > > 
> > > > > > > > > > > > To be able to provide highlighting snippets for large textual  
> > > > > > > > > > > > ('content') field, it needs to be stored.
> > > > > > > > > > > > 
> > > > > > > > > > > > Now we are in situation where because textual field is too big  
> > > > > > > > > > > > (makes  
> > > > > > > > > > > > impassible to return json source for 30-50 results to a client)  
> > > > > > > > > > > > we need  
> > > > > > > > > > > > to store it twice in index (once as a part of original json  
> > > > > > > > > > > > source in  
> > > > > > > > > > > > \_source and second time as a 'content' field "store" : "yes" for  
> > > > > > > > > > > > highlighting). This make index a lot bigger.
> > > > > > > > > > > > 
> > > > > > > > > > > > Also, if there is a requirement to display highlighting for all  
> > > > > > > > > > > > fields  
> > > > > > > > > > > > (separate highlight snippets for each field where match  
> > > > > > > > > > > > occurred,  
> > > > > > > > > > > > without mixing fields snippets -\> stored \_all field can not be  
> > > > > > > > > > > > used for  
> > > > > > > > > > > > highlighting in such case) then whole document (all fields)  
> > > > > > > > > > > > needs to be  
> > > > > > > > > > > > stored twice.
> > > > > > > > > > > > 
> > > > > > > > > > > > In this case seems only logical to disable \_source field (since  
> > > > > > > > > > > > all  
> > > > > > > > > > > > fields are stored anyway) and when whole document needs to be  
> > > > > > > > > > > > retrieved  
> > > > > > > > > > > > use (newly added) fields=\* feature (I've read similar discussion  
> > > > > > > > > > > > thread  
> > > > > > > > > > > > which led to this enhancement)
> > > > > > > > > > > > 
> > > > > > > > > > > > Here is my question/proposal:
> > > > > > > > > > > > 
> > > > > > > > > > > > would it be possible to enable use of json \_source field for  
> > > > > > > > > > > > 'field  
> > > > > > > > > > > > specific' highlighting, where matching snippet needs to be  
> > > > > > > > > > > > returned  
> > > > > > > > > > > > separately for each field name?
> > > > > > > > > > > > 
> > > > > > > > > > > > Maybe to have term\_vector for each field, but to somehow  
> > > > > > > > > > > > 'adjust' or  
> > > > > > > > > > > > recalculate positions\_offsets to point to text snippet in  
> > > > > > > > > > > > \_source  
> > > > > > > > > > > > instead of stored field?
> > > > > > > > > > > > 
> > > > > > > > > > > > I know this is not a simple requirement, but if 'field specific'  
> > > > > > > > > > > > highlighting could somehow use stored json instead of requiring  
> > > > > > > > > > > > an  
> > > > > > > > > > > > individual field to be (separately) stored, that would make a  
> > > > > > > > > > > > great  
> > > > > > > > > > > > use/reuse of stored json \_source (and no one would ever think of  
> > > > > > > > > > > > disabling it 🙂
> > > > > > > > > > > > 
> > > > > > > > > > > > Tomislav

---

<div class="post-metadata">

**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\
**Post date:** [August 13, 2010, 8:21am UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213/14 "2010-08-13T08:21:31Z")

</div>

On Fri, Aug 13, 2010 at 10:20 AM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote:

> Yes, I did not realize that earlier but you are right that I will need to  
> pass query into the highlight section as well.  
> Take the following example:
> 
> I need to display all candidates that match "dude java" query and then I  
> want to allow user to click on individual name and get whole bio  
> highlighted.
> 
> So how I can go about this:  
> First, I can get relevant documents using simple "query\_string" query for  
> "dude java". I can now display names of candidates without highlights and  
> highlighted fragments from bio for each name, kind of basic search interface  
> that already works now. But if I wanted to display highlighted name I would  
> get something like "..e _Dude_ Abid..." which is not what I want  
> (sure, I can work with fragment size but that is just workaround and does  
> not fit all situations). So when using that "query\_string" query I would  
> like to specify in the highlight section that the person.name should be  
> highlighted with no fragments.
> 
> Second, now, when the user clicks individual name, then I want to get whole  
> bio highlighted.  
> So I need to get specific document (by ID) and have the bio  
> field highlighted (and the name field as well)  
> The example of the query that could be used:
> 
> curl -XGET [http://localhost:9200/\_all/\_search](http://localhost:9200/_all/_search) -d '  
> { "query" : { "term" : { "person-id" : "1234" } },  
> "highlight" : {  
> "fields" : {  
> "\_source" : {  
> "path" : "person.bio,person.name",  
> "fragmenter" : "classpath.to.NullFragmenter",  
> "query" : {  
> "query\_string" : { "fields" : ["bio","name"], "query" : "dude  
> java" }  
> }  
> }  
> }  
> }  
> }'
> 
> or I could use fields query:
> 
> curl -XGET [http://localhost:9200/\_all/\_search](http://localhost:9200/_all/_search) -d '  
> { "query" : { "term" : { "person-id" : "1234" } },  
> "fields" : ["bio","name"],  
> "highlight" : {  
> "fields" : {  
> "bio" : {  
> "query" : {  
> "query\_string" : { "fields" : ["bio"], "query" : "dude java" }  
> }  
> },  
> "name" : {  
> "query" : {  
> "query\_string" : { "fields" : ["name"], "query" : "dude java" }  
> }  
> }  
> }  
> }  
> }'
> 
> The later query requires both bio and name to be stored (and this is where  
> it gets back to Tomislav's original point I think).  
> Ugh! I am complicating it way too much... but hope the request is clear now  
> 🙂

Sure I am complicating it too much because in the later query example I  
forgot the specify NullFragmenter 🙂

> Regards,  
> Lukas
> 
> 2010/8/13 Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> 
> ok, so you want to get the whole bio field highlighted, so you would need
> 
> > to pass the query to the get API as well, otherwise, there is no way to  
> > highlight it (aside from other things you need, like the option to do no  
> > fragmentation and getting the actual data).
> > 
> > On Thu, Aug 12, 2010 at 10:34 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > 
> > > Oh, and one more note, see below:
> > > 
> > > On Thu, Aug 12, 2010 at 9:22 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > 
> > > > If I want to display whole bio highlighted then I can either get  
> > > > "\_source" and cut bio from it on the client side but in this case I need to  
> > > > tell ES to use highlighting on it first. Or I need to specify in mapping  
> > > > that bio is also stored and use fields query  
> > > > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/search/fields/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/search/fields/) but  
> > > > again I need to tell ES to highlight it. And in neither case I want only  
> > > > fragments, I want WHOLE content of the field. The first approach is not  
> > > > possible now the later is possible but required bio to be explicitly stored  
> > > > (and it is already stored in \_source).
> > > 
> > > And the later also requires specification of Fragmenter that returns  
> > > whole body, not fragments, thus my reference to NullFragmenter, which is not  
> > > implemented in FastVectorHighlighter API (as far as I understand it), it can  
> > > be found in the older Highlighting API, thus I opened also  
> > > [Highlighting API: Add support for custom FragmentsBuilder · Issue #307 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/307)
> > > 
> > > May be it would be better if the NullFragmenter-like functionality is  
> > > contributed directly into Lucene FastVectorHighlighter API. I was looking at  
> > > the FVH API today and I think I can try to implement such Fragmenter.
> > > 
> > > > Hope this makes it clear. (Sorry if I confused you).
> > > > 
> > > > Lukas
> > > > 
> > > > 2010/8/12 Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> > > > 
> > > > > Ahh, I see. So you would still need to provide a query to the GET api  
> > > > > in order to do the highlighting, right?
> > > > > 
> > > > > On Thu, Aug 12, 2010 at 10:08 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > > > 
> > > > > > Imagine search app for HR: Candidate catalog (cool name!).  
> > > > > > The entities stored in the index are as follows: person: { id, name,  
> > > > > > address, bio }  
> > > > > > Now I am using just the REST API. Say I search for "Java" and I would  
> > > > > > like to display list of Names and allow users to click individual name which  
> > > > > > would display whole bio with Java highlighted in it (here comes the  
> > > > > > highlighting in the play!). Now I can display bio (just using GET REST API  
> > > > > > with given document ID but not highlighted. So I was thinking that it would  
> > > > > > be cool to have this function.
> > > > > > 
> > > > > > Lukas
> > > > > > 
> > > > > > 2010/8/12 Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> > > > > > 
> > > > > > So, what you want is to be able to get just the bio field, without the
> > > > > > 
> > > > > > > full source, and without the bio field being stored? If so, then the  
> > > > > > > response I gave, where the logic might apply also to get fields using  
> > > > > > > something like "source\_field" notion applies here. It does mean that the  
> > > > > > > full source will need to be retrieved and parsed. Not sure how highlighting  
> > > > > > > comes into play here...
> > > > > > > 
> > > > > > > -shay.banon
> > > > > > > 
> > > > > > > On Thu, Aug 12, 2010 at 9:53 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > > > > > 
> > > > > > > > Actually, that ticket has two parts. One is Fragmenter related and  
> > > > > > > > the other one is possibility to tell, that I want to highlight some portion  
> > > > > > > > of \_source data. Imagine I am using only REST API and for example if \_source  
> > > > > > > > is a person with name, address and bio fields then I would like to tell that  
> > > > > > > > I want to highlight just the bio field (and I think the NullFragmenter would  
> > > > > > > > be needed for this if I want to display whole content of bio highlighted,  
> > > > > > > > not just fragments). The other possibility would be to define mapping for  
> > > > > > > > person in such a way that bio would be a stored field, then I could query  
> > > > > > > > for stored fields (not pulling the \_source field) and tell the I want to  
> > > > > > > > apply NullFragmenter to this data while highlighting. But this gets back to  
> > > > > > > > the Tomislav's situation, because this would mean that bio is probably  
> > > > > > > > stored twice, once as a part of source and then separately as a stored bio  
> > > > > > > > field.
> > > > > > > > 
> > > > > > > > Lukas
> > > > > > > > 
> > > > > > > > On Thu, Aug 12, 2010 at 8:41 PM, Shay Banon \<  
> > > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > > > 
> > > > > > > > > Not sure if it overlaps, fragmenter controls how to break the  
> > > > > > > > > highlighted data, this relates to how to fetch that date to highlight.
> > > > > > > > > 
> > > > > > > > > -shay.banon
> > > > > > > > > 
> > > > > > > > > On Thu, Aug 12, 2010 at 9:08 PM, Lukáš Vlček \<  
> > > > > > > > > [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)\> wrote:
> > > > > > > > > 
> > > > > > > > > > One of differences is that the 308 issue was meant to return whole  
> > > > > > > > > > content of the \_source or some of its fields (or stored fields if not using  
> > > > > > > > > > "\_source"). But the point is that the user should be able to specify  
> > > > > > > > > > Fragmenter type (or provide custom implementation of Fragmenter).
> > > > > > > > > > 
> > > > > > > > > > On Thu, Aug 12, 2010 at 8:03 PM, Lukáš Vlček \<  
> > > > > > > > > > [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)\> wrote:
> > > > > > > > > > 
> > > > > > > > > > > If I read it correctly then I think it partly overlaps with  
> > > > > > > > > > > [Highlighting API: Apply highlighting to REST GET operation · Issue #308 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/308)
> > > > > > > > > > > 
> > > > > > > > > > > [http://github.com/elasticsearch/elasticsearch/issues/issue/308](http://github.com/elasticsearch/elasticsearch/issues/issue/308)  
> > > > > > > > > > > Lukas
> > > > > > > > > > > 
> > > > > > > > > > > On Thu, Aug 12, 2010 at 5:53 PM, Shay Banon \<  
> > > > > > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > Agreed. A bit tricky to implement, but possible. Also, note that  
> > > > > > > > > > > > this will require loading the full json from the index, and parse it in  
> > > > > > > > > > > > order to get the relevant parts from it. It won't be returned, but still  
> > > > > > > > > > > > loaded. So, it might not make sense when you have several big fields in the  
> > > > > > > > > > > > index, and you want to get fragments for one of them. But does make sense  
> > > > > > > > > > > > when having one big field.
> > > > > > > > > > > > 
> > > > > > > > > > > > Also, if this is implemented, it should also be possible to get  
> > > > > > > > > > > > specific fields out of the json as a response as well (similar to asking for  
> > > > > > > > > > > > specific fields in the search request, maybe call them source\_fileds).
> > > > > > > > > > > > 
> > > > > > > > > > > > Open an issue for this?
> > > > > > > > > > > > 
> > > > > > > > > > > > -shay.banon
> > > > > > > > > > > > 
> > > > > > > > > > > > On Thu, Aug 12, 2010 at 6:06 PM, Tomislav Poljak \<  
> > > > > > > > > > > > [tpoljak@gmail.com](mailto:tpoljak@gmail.com)\> wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > > Hi,  
> > > > > > > > > > > > > I really like all the features stored json (enabled \_source)  
> > > > > > > > > > > > > provides in  
> > > > > > > > > > > > > both Java API (used it in indexing/updating) and REST API (used  
> > > > > > > > > > > > > it for  
> > > > > > > > > > > > > searching).
> > > > > > > > > > > > > 
> > > > > > > > > > > > > I do however have one possible request for improvement  
> > > > > > > > > > > > > regarding  
> > > > > > > > > > > > > documents with large textual fields and overall highlighting.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > When there is a requirement to index/search documents with  
> > > > > > > > > > > > > large textual  
> > > > > > > > > > > > > fields (like 'content' with text in Mb, which is not unusual),  
> > > > > > > > > > > > > returning  
> > > > > > > > > > > > > a whole json for each result in result set can be impossible  
> > > > > > > > > > > > > (if each  
> > > > > > > > > > > > > json document has a few Mb in 'content' returning 30-50 results  
> > > > > > > > > > > > > to a  
> > > > > > > > > > > > > client doesn't sound realistic or even possible in acceptable  
> > > > > > > > > > > > > time with  
> > > > > > > > > > > > > usual 'Internet' bandwidth).
> > > > > > > > > > > > > 
> > > > > > > > > > > > > But, usually it's acceptable (or even requirement) to  
> > > > > > > > > > > > > display/return  
> > > > > > > > > > > > > only highlighting snippets for 30-50 results matches and  
> > > > > > > > > > > > > retrieve whole  
> > > > > > > > > > > > > document (json source) only for a single document (when  
> > > > > > > > > > > > > requested by  
> > > > > > > > > > > > > exact ID).
> > > > > > > > > > > > > 
> > > > > > > > > > > > > To be able to provide highlighting snippets for large textual  
> > > > > > > > > > > > > ('content') field, it needs to be stored.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Now we are in situation where because textual field is too big  
> > > > > > > > > > > > > (makes  
> > > > > > > > > > > > > impassible to return json source for 30-50 results to a client)  
> > > > > > > > > > > > > we need  
> > > > > > > > > > > > > to store it twice in index (once as a part of original json  
> > > > > > > > > > > > > source in  
> > > > > > > > > > > > > \_source and second time as a 'content' field "store" : "yes"  
> > > > > > > > > > > > > for  
> > > > > > > > > > > > > highlighting). This make index a lot bigger.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Also, if there is a requirement to display highlighting for all  
> > > > > > > > > > > > > fields  
> > > > > > > > > > > > > (separate highlight snippets for each field where match  
> > > > > > > > > > > > > occurred,  
> > > > > > > > > > > > > without mixing fields snippets -\> stored \_all field can not be  
> > > > > > > > > > > > > used for  
> > > > > > > > > > > > > highlighting in such case) then whole document (all fields)  
> > > > > > > > > > > > > needs to be  
> > > > > > > > > > > > > stored twice.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > In this case seems only logical to disable \_source field (since  
> > > > > > > > > > > > > all  
> > > > > > > > > > > > > fields are stored anyway) and when whole document needs to be  
> > > > > > > > > > > > > retrieved  
> > > > > > > > > > > > > use (newly added) fields=\* feature (I've read similar  
> > > > > > > > > > > > > discussion thread  
> > > > > > > > > > > > > which led to this enhancement)
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Here is my question/proposal:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > would it be possible to enable use of json \_source field for  
> > > > > > > > > > > > > 'field  
> > > > > > > > > > > > > specific' highlighting, where matching snippet needs to be  
> > > > > > > > > > > > > returned  
> > > > > > > > > > > > > separately for each field name?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Maybe to have term\_vector for each field, but to somehow  
> > > > > > > > > > > > > 'adjust' or  
> > > > > > > > > > > > > recalculate positions\_offsets to point to text snippet in  
> > > > > > > > > > > > > \_source  
> > > > > > > > > > > > > instead of stored field?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > I know this is not a simple requirement, but if 'field  
> > > > > > > > > > > > > specific'  
> > > > > > > > > > > > > highlighting could somehow use stored json instead of requiring  
> > > > > > > > > > > > > an  
> > > > > > > > > > > > > individual field to be (separately) stored, that would make a  
> > > > > > > > > > > > > great  
> > > > > > > > > > > > > use/reuse of stored json \_source (and no one would ever think  
> > > > > > > > > > > > > of  
> > > > > > > > > > > > > disabling it 🙂
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Tomislav

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 13, 2010, 9:50am UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213/15 "2010-08-13T09:50:32Z")

</div>

No problem, I understand the general idea of the requirement.

-shay.banon

On Fri, Aug 13, 2010 at 11:21 AM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote:

> On Fri, Aug 13, 2010 at 10:20 AM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> 
> > Yes, I did not realize that earlier but you are right that I will need to  
> > pass query into the highlight section as well.  
> > Take the following example:
> > 
> > I need to display all candidates that match "dude java" query and then I  
> > want to allow user to click on individual name and get whole bio  
> > highlighted.
> > 
> > So how I can go about this:  
> > First, I can get relevant documents using simple "query\_string" query for  
> > "dude java". I can now display names of candidates without highlights and  
> > highlighted fragments from bio for each name, kind of basic search interface  
> > that already works now. But if I wanted to display highlighted name I would  
> > get something like "..e _Dude_ Abid..." which is not what I want  
> > (sure, I can work with fragment size but that is just workaround and does  
> > not fit all situations). So when using that "query\_string" query I would  
> > like to specify in the highlight section that the person.name should be  
> > highlighted with no fragments.
> > 
> > Second, now, when the user clicks individual name, then I want to get  
> > whole bio highlighted.  
> > So I need to get specific document (by ID) and have the bio  
> > field highlighted (and the name field as well)  
> > The example of the query that could be used:
> > 
> > curl -XGET [http://localhost:9200/\_all/\_search](http://localhost:9200/_all/_search) -d '  
> > { "query" : { "term" : { "person-id" : "1234" } },  
> > "highlight" : {  
> > "fields" : {  
> > "\_source" : {  
> > "path" : "person.bio,person.name",  
> > "fragmenter" : "classpath.to.NullFragmenter",  
> > "query" : {  
> > "query\_string" : { "fields" : ["bio","name"], "query" : "dude  
> > java" }  
> > }  
> > }  
> > }  
> > }  
> > }'
> > 
> > or I could use fields query:
> > 
> > curl -XGET [http://localhost:9200/\_all/\_search](http://localhost:9200/_all/_search) -d '  
> > { "query" : { "term" : { "person-id" : "1234" } },  
> > "fields" : ["bio","name"],  
> > "highlight" : {  
> > "fields" : {  
> > "bio" : {  
> > "query" : {  
> > "query\_string" : { "fields" : ["bio"], "query" : "dude java" }  
> > }  
> > },  
> > "name" : {  
> > "query" : {  
> > "query\_string" : { "fields" : ["name"], "query" : "dude java" }  
> > }  
> > }  
> > }  
> > }  
> > }'
> > 
> > The later query requires both bio and name to be stored (and this is where  
> > it gets back to Tomislav's original point I think).  
> > Ugh! I am complicating it way too much... but hope the request is clear  
> > now 🙂
> 
> Sure I am complicating it too much because in the later query example I  
> forgot the specify NullFragmenter 🙂
> 
> > Regards,  
> > Lukas
> > 
> > 2010/8/13 Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> > 
> > ok, so you want to get the whole bio field highlighted, so you would need
> > 
> > > to pass the query to the get API as well, otherwise, there is no way to  
> > > highlight it (aside from other things you need, like the option to do no  
> > > fragmentation and getting the actual data).
> > > 
> > > On Thu, Aug 12, 2010 at 10:34 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > 
> > > > Oh, and one more note, see below:
> > > > 
> > > > On Thu, Aug 12, 2010 at 9:22 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > > 
> > > > > If I want to display whole bio highlighted then I can either get  
> > > > > "\_source" and cut bio from it on the client side but in this case I need to  
> > > > > tell ES to use highlighting on it first. Or I need to specify in mapping  
> > > > > that bio is also stored and use fields query  
> > > > > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/search/fields/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/search/fields/) but  
> > > > > again I need to tell ES to highlight it. And in neither case I want only  
> > > > > fragments, I want WHOLE content of the field. The first approach is not  
> > > > > possible now the later is possible but required bio to be explicitly stored  
> > > > > (and it is already stored in \_source).
> > > > 
> > > > And the later also requires specification of Fragmenter that returns  
> > > > whole body, not fragments, thus my reference to NullFragmenter, which is not  
> > > > implemented in FastVectorHighlighter API (as far as I understand it), it can  
> > > > be found in the older Highlighting API, thus I opened also  
> > > > [Highlighting API: Add support for custom FragmentsBuilder · Issue #307 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/307)
> > > > 
> > > > May be it would be better if the NullFragmenter-like functionality is  
> > > > contributed directly into Lucene FastVectorHighlighter API. I was looking at  
> > > > the FVH API today and I think I can try to implement such Fragmenter.
> > > > 
> > > > > Hope this makes it clear. (Sorry if I confused you).
> > > > > 
> > > > > Lukas
> > > > > 
> > > > > 2010/8/12 Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> > > > > 
> > > > > > Ahh, I see. So you would still need to provide a query to the GET api  
> > > > > > in order to do the highlighting, right?
> > > > > > 
> > > > > > On Thu, Aug 12, 2010 at 10:08 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> > > > > > 
> > > > > > > Imagine search app for HR: Candidate catalog (cool name!).  
> > > > > > > The entities stored in the index are as follows: person: { id, name,  
> > > > > > > address, bio }  
> > > > > > > Now I am using just the REST API. Say I search for "Java" and I would  
> > > > > > > like to display list of Names and allow users to click individual name which  
> > > > > > > would display whole bio with Java highlighted in it (here comes the  
> > > > > > > highlighting in the play!). Now I can display bio (just using GET REST API  
> > > > > > > with given document ID but not highlighted. So I was thinking that it would  
> > > > > > > be cool to have this function.
> > > > > > > 
> > > > > > > Lukas
> > > > > > > 
> > > > > > > 2010/8/12 Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> > > > > > > 
> > > > > > > So, what you want is to be able to get just the bio field, without
> > > > > > > 
> > > > > > > > the full source, and without the bio field being stored? If so, then the  
> > > > > > > > response I gave, where the logic might apply also to get fields using  
> > > > > > > > something like "source\_field" notion applies here. It does mean that the  
> > > > > > > > full source will need to be retrieved and parsed. Not sure how highlighting  
> > > > > > > > comes into play here...
> > > > > > > > 
> > > > > > > > -shay.banon
> > > > > > > > 
> > > > > > > > On Thu, Aug 12, 2010 at 9:53 PM, Lukáš Vlček \<[lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)
> > > > > > > > 
> > > > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > Actually, that ticket has two parts. One is Fragmenter related and  
> > > > > > > > > the other one is possibility to tell, that I want to highlight some portion  
> > > > > > > > > of \_source data. Imagine I am using only REST API and for example if \_source  
> > > > > > > > > is a person with name, address and bio fields then I would like to tell that  
> > > > > > > > > I want to highlight just the bio field (and I think the NullFragmenter would  
> > > > > > > > > be needed for this if I want to display whole content of bio highlighted,  
> > > > > > > > > not just fragments). The other possibility would be to define mapping for  
> > > > > > > > > person in such a way that bio would be a stored field, then I could query  
> > > > > > > > > for stored fields (not pulling the \_source field) and tell the I want to  
> > > > > > > > > apply NullFragmenter to this data while highlighting. But this gets back to  
> > > > > > > > > the Tomislav's situation, because this would mean that bio is probably  
> > > > > > > > > stored twice, once as a part of source and then separately as a stored bio  
> > > > > > > > > field.
> > > > > > > > > 
> > > > > > > > > Lukas
> > > > > > > > > 
> > > > > > > > > On Thu, Aug 12, 2010 at 8:41 PM, Shay Banon \<  
> > > > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > > > > 
> > > > > > > > > > Not sure if it overlaps, fragmenter controls how to break the  
> > > > > > > > > > highlighted data, this relates to how to fetch that date to highlight.
> > > > > > > > > > 
> > > > > > > > > > -shay.banon
> > > > > > > > > > 
> > > > > > > > > > On Thu, Aug 12, 2010 at 9:08 PM, Lukáš Vlček \<  
> > > > > > > > > > [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)\> wrote:
> > > > > > > > > > 
> > > > > > > > > > > One of differences is that the 308 issue was meant to return  
> > > > > > > > > > > whole content of the \_source or some of its fields (or stored fields if not  
> > > > > > > > > > > using "\_source"). But the point is that the user should be able to specify  
> > > > > > > > > > > Fragmenter type (or provide custom implementation of Fragmenter).
> > > > > > > > > > > 
> > > > > > > > > > > On Thu, Aug 12, 2010 at 8:03 PM, Lukáš Vlček \<  
> > > > > > > > > > > [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)\> wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > If I read it correctly then I think it partly overlaps with  
> > > > > > > > > > > > [Highlighting API: Apply highlighting to REST GET operation · Issue #308 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/308)
> > > > > > > > > > > > 
> > > > > > > > > > > > [http://github.com/elasticsearch/elasticsearch/issues/issue/308](http://github.com/elasticsearch/elasticsearch/issues/issue/308)  
> > > > > > > > > > > > Lukas
> > > > > > > > > > > > 
> > > > > > > > > > > > On Thu, Aug 12, 2010 at 5:53 PM, Shay Banon \<  
> > > > > > > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > > Agreed. A bit tricky to implement, but possible. Also, note  
> > > > > > > > > > > > > that this will require loading the full json from the index, and parse it in  
> > > > > > > > > > > > > order to get the relevant parts from it. It won't be returned, but still  
> > > > > > > > > > > > > loaded. So, it might not make sense when you have several big fields in the  
> > > > > > > > > > > > > index, and you want to get fragments for one of them. But does make sense  
> > > > > > > > > > > > > when having one big field.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Also, if this is implemented, it should also be possible to get  
> > > > > > > > > > > > > specific fields out of the json as a response as well (similar to asking for  
> > > > > > > > > > > > > specific fields in the search request, maybe call them source\_fileds).
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Open an issue for this?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > -shay.banon
> > > > > > > > > > > > > 
> > > > > > > > > > > > > On Thu, Aug 12, 2010 at 6:06 PM, Tomislav Poljak \<  
> > > > > > > > > > > > > [tpoljak@gmail.com](mailto:tpoljak@gmail.com)\> wrote:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > Hi,  
> > > > > > > > > > > > > > I really like all the features stored json (enabled \_source)  
> > > > > > > > > > > > > > provides in  
> > > > > > > > > > > > > > both Java API (used it in indexing/updating) and REST API  
> > > > > > > > > > > > > > (used it for  
> > > > > > > > > > > > > > searching).
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > I do however have one possible request for improvement  
> > > > > > > > > > > > > > regarding  
> > > > > > > > > > > > > > documents with large textual fields and overall highlighting.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > When there is a requirement to index/search documents with  
> > > > > > > > > > > > > > large textual  
> > > > > > > > > > > > > > fields (like 'content' with text in Mb, which is not unusual),  
> > > > > > > > > > > > > > returning  
> > > > > > > > > > > > > > a whole json for each result in result set can be impossible  
> > > > > > > > > > > > > > (if each  
> > > > > > > > > > > > > > json document has a few Mb in 'content' returning 30-50  
> > > > > > > > > > > > > > results to a  
> > > > > > > > > > > > > > client doesn't sound realistic or even possible in acceptable  
> > > > > > > > > > > > > > time with  
> > > > > > > > > > > > > > usual 'Internet' bandwidth).
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > But, usually it's acceptable (or even requirement) to  
> > > > > > > > > > > > > > display/return  
> > > > > > > > > > > > > > only highlighting snippets for 30-50 results matches and  
> > > > > > > > > > > > > > retrieve whole  
> > > > > > > > > > > > > > document (json source) only for a single document (when  
> > > > > > > > > > > > > > requested by  
> > > > > > > > > > > > > > exact ID).
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > To be able to provide highlighting snippets for large textual  
> > > > > > > > > > > > > > ('content') field, it needs to be stored.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Now we are in situation where because textual field is too big  
> > > > > > > > > > > > > > (makes  
> > > > > > > > > > > > > > impassible to return json source for 30-50 results to a  
> > > > > > > > > > > > > > client) we need  
> > > > > > > > > > > > > > to store it twice in index (once as a part of original json  
> > > > > > > > > > > > > > source in  
> > > > > > > > > > > > > > \_source and second time as a 'content' field "store" : "yes"  
> > > > > > > > > > > > > > for  
> > > > > > > > > > > > > > highlighting). This make index a lot bigger.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Also, if there is a requirement to display highlighting for  
> > > > > > > > > > > > > > all fields  
> > > > > > > > > > > > > > (separate highlight snippets for each field where match  
> > > > > > > > > > > > > > occurred,  
> > > > > > > > > > > > > > without mixing fields snippets -\> stored \_all field can not be  
> > > > > > > > > > > > > > used for  
> > > > > > > > > > > > > > highlighting in such case) then whole document (all fields)  
> > > > > > > > > > > > > > needs to be  
> > > > > > > > > > > > > > stored twice.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > In this case seems only logical to disable \_source field  
> > > > > > > > > > > > > > (since all  
> > > > > > > > > > > > > > fields are stored anyway) and when whole document needs to be  
> > > > > > > > > > > > > > retrieved  
> > > > > > > > > > > > > > use (newly added) fields=\* feature (I've read similar  
> > > > > > > > > > > > > > discussion thread  
> > > > > > > > > > > > > > which led to this enhancement)
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Here is my question/proposal:
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > would it be possible to enable use of json \_source field for  
> > > > > > > > > > > > > > 'field  
> > > > > > > > > > > > > > specific' highlighting, where matching snippet needs to be  
> > > > > > > > > > > > > > returned  
> > > > > > > > > > > > > > separately for each field name?
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Maybe to have term\_vector for each field, but to somehow  
> > > > > > > > > > > > > > 'adjust' or  
> > > > > > > > > > > > > > recalculate positions\_offsets to point to text snippet in  
> > > > > > > > > > > > > > \_source  
> > > > > > > > > > > > > > instead of stored field?
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > I know this is not a simple requirement, but if 'field  
> > > > > > > > > > > > > > specific'  
> > > > > > > > > > > > > > highlighting could somehow use stored json instead of  
> > > > > > > > > > > > > > requiring an  
> > > > > > > > > > > > > > individual field to be (separately) stored, that would make a  
> > > > > > > > > > > > > > great  
> > > > > > > > > > > > > > use/reuse of stored json \_source (and no one would ever think  
> > > > > > > > > > > > > > of  
> > > > > > > > > > > > > > disabling it 🙂
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Tomislav

---

<div class="post-metadata">

**Author:** ![Tomislav\_Poljak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomislav_poljak/32/1177_2.png) [@Tomislav\_Poljak](https://discuss.elastic.co/u/Tomislav_Poljak)\
**Post date:** [August 13, 2010, 2:30pm UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213/16 "2010-08-13T14:30:24Z")

</div>

Hi,  
I'm not sure I fully understand what will be implement as a  
result/conclusion of discussion here, but I think I can define what I  
would like to be implemented (from my point of view) pretty clearly as:

It would be great if ES, beside returning whole document source (in json  
format) in search results, supported returning json type structure with  
'matching fields' and/or requested fields. Only fields which are matched  
by a query or requested would be returned (from \_source json) and this  
would be possible without storing each field separately. Value in these  
fields would be either a whole field value (with highlighting applied)  
or a highlighting snippet (for large textual fields).

Will something like that be possible?

Thanks,  
Tomislav

On Fri, 2010-08-13 at 12:50 +0300, Shay Banon wrote:

> No problem, I understand the general idea of the requirement.
> 
> -shay.banon
> 
> On Fri, Aug 13, 2010 at 11:21 AM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)  
> wrote:
> 
> ```
> On Fri, Aug 13, 2010 at 10:20 AM, Lukáš Vlček
> <lukas.vlcek@gmail.com> wrote:
> Yes, I did not realize that earlier but you are right
> that I will need to pass query into the highlight
> section as well.
> Take the following example:
>             
>             
> I need to display all candidates that match "dude
> java" query and then I want to allow user to click on
> individual name and get whole bio highlighted.
>             
>             
> So how I can go about this:
> First, I can get relevant documents using simple
> "query_string" query for "dude java". I can now
> display names of candidates without highlights and
> highlighted fragments from bio for each name, kind of
> basic search interface that already works now. But if
> I wanted to display highlighted name I would get
> something like "..e <em>Dude</em> Abid..." which is
> not what I want (sure, I can work with fragment size
> but that is just workaround and does not fit all
> situations). So when using that "query_string" query I
> would like to specify in the highlight section that
> the person.name should be highlighted with no
> fragments.
>             
>             
> Second, now, when the user clicks individual name,
> then I want to get whole bio highlighted.
> So I need to get specific document (by ID) and have
> the bio field highlighted (and the name field as well)
> The example of the query that could be used:
>             
> curl -XGET http://localhost:9200/_all/_search -d '
> { "query" : { "term" : { "person-id" : "1234" } },
> "highlight" : {
> "fields" : {
> "_source" : {
> "path" : "person.bio,person.name",
> "fragmenter" : "classpath.to.NullFragmenter",
> "query" : {
> "query_string" : { "fields" :
> ["bio","name"], "query" : "dude java" }
> }
> }
> }
> }
> }'
>             
>             
> or I could use fields query:
>             
>             
> curl -XGET http://localhost:9200/_all/_search -d '
> { "query" : { "term" : { "person-id" : "1234" } },
> "fields" : ["bio","name"],
> "highlight" : {
> "fields" : {
> "bio" : {
> "query" : {
> "query_string" : { "fields" : ["bio"],
> "query" : "dude java" }
> }
> },
> "name" : {
> "query" : {
> "query_string" : { "fields" : ["name"],
> "query" : "dude java" }
> }
> }
> }
> }
> }'
>             
>             
> The later query requires both bio and name to be
> stored (and this is where it gets back to Tomislav's
> original point I think).
> Ugh! I am complicating it way too much... but hope the
> request is clear now :-)
>     
>     
> Sure I am complicating it too much because in the later query
> example I forgot the specify NullFragmenter :-)
>     
>      
>             
>             
> Regards,
> Lukas
>             
>             
>             
> 2010/8/13 Shay Banon <shay.banon@elasticsearch.com>
>             
>             
> ok, so you want to get the whole bio field
> highlighted, so you would need to pass the
> query to the get API as well, otherwise, there
> is no way to highlight it (aside from other
> things you need, like the option to do no
> fragmentation and getting the actual data).
>                     
>                     
>                     
> On Thu, Aug 12, 2010 at 10:34 PM, Lukáš Vlček
> <lukas.vlcek@gmail.com> wrote:
> Oh, and one more note, see below:
>                             
> On Thu, Aug 12, 2010 at 9:22 PM, Lukáš
> Vlček <lukas.vlcek@gmail.com> wrote:
> If I want to display whole bio
> highlighted then I can either
> get "_source" and cut bio from
> it on the client side but in
> this case I need to tell ES to
> use highlighting on it first.
> Or I need to specify in
> mapping that bio is also
> stored and use fields
> query http://www.elasticsearch.com/docs/elasticsearch/rest_api/search/fields/ but again I need to tell ES to highlight it. And in neither case I want only fragments, I want WHOLE content of the field. The first approach is not possible now the later is possible but required bio to be explicitly stored (and it is already stored in _source).
>                             
>                             
> And the later also requires
> specification of Fragmenter that
> returns whole body, not fragments,
> thus my reference to NullFragmenter,
> which is not implemented in
> FastVectorHighlighter API (as far as I
> understand it), it can be found in the
> older Highlighting API, thus I opened
> also http://github.com/elasticsearch/elasticsearch/issues/issue/307 
>                             
>                             
> May be it would be better if the
> NullFragmenter-like functionality is
> contributed directly into Lucene
> FastVectorHighlighter API. I was
> looking at the FVH API today and I
> think I can try to implement such
> Fragmenter.
>                             
>                             
>                             
>                                     
>                                     
> Hope this makes it clear.
> (Sorry if I confused you).
>                                     
>                                     
>                                     
> Lukas
>                                     
> 2010/8/12 Shay Banon
> <shay.banon@elasticsearch.com>
> Ahh, I see. So you
> would still need to
> provide a query to the
> GET api in order to do
> the highlighting,
> right?
>                                             
>                                             
>                                             
> On Thu, Aug 12, 2010
> at 10:08 PM, Lukáš
> Vlček
> <lukas.vlcek@gmail.com> wrote:
> Imagine search
> app for HR:
> Candidate
> catalog (cool
> name!).
> The entities
> stored in the
> index are as
> follows:
> person: { id,
> name, address,
> bio }
> Now I am using
> just the REST
> API. Say I
> search for
> "Java" and I
> would like to
> display list
> of Names and
> allow users to
> click
> individual
> name which
> would display
> whole bio with
> Java
> highlighted in
> it (here comes
> the
> highlighting
> in the play!).
> Now I can
> display bio
> (just using
> GET REST API
> with given
> document ID
> but not
> highlighted.
> So I was
> thinking that
> it would be
> cool to have
> this function.
>                                                     
>                                                     
> Lukas
>                                                     
> 2010/8/12 Shay
> Banon
> <shay.banon@elasticsearch.com>
>                                                     
>                                                     
> So,
> what
> you
> want
> is to
> be
> able
> to get
> just
> the
> bio
> field,
> without the full source, and without the bio field being stored? If so, then the response I gave, where the logic might apply also to get fields using something like "source_field" notion applies here. It does mean that the full source will need to be retrieved and parsed. Not sure how highlighting comes into play here...
>                                                             
>                                                             
> -shay.banon
>                                                             
>                                                             
>                                                             
> On
> Thu,
> Aug
> 12,
> 2010
> at
> 9:53
> PM,
> Lukáš
> Vlček
> <lukas.vlcek@gmail.com> wrote:
> Actually, that ticket has two parts. One is Fragmenter related and the other one is possibility to tell, that I want to highlight some portion of _source data. Imagine I am using only REST API and for example if _source is a person with name, address and bio fields then I would like to tell that I want to highlight just the bio field (and I think the NullFragmenter would be needed for this if I want to display whole content of bio highlighted, not just fragments). The other possibility would be to define mapping for person in such a way that bio would be a stored field, then I could query for stored fields (not pulling the _source field) and tell the I want to apply NullFragmenter to this data while highlighting. But this gets back to the Tomislav's situation, because this would mean that bio is probably stored twice, once as a part of source and then separately as a stored bio field.
>                                                                     
>                                                                     
> Lukas
>                                                                     
>                                                                     
>                                                                     
> On Thu, Aug 12, 2010 at 8:41 PM, Shay Banon <shay.banon@elasticsearch.com> wrote:
> Not sure if it overlaps, fragmenter controls how to break the highlighted data, this relates to how to fetch that date to highlight.
>                                                                             
>                                                                             
> -shay.banon
>                                                                             
>                                                                             
>                                                                             
> On Thu, Aug 12, 2010 at 9:08 PM, Lukáš Vlček <lukas.vlcek@gmail.com> wrote:
> One of differences is that the 308 issue was meant to return whole content of the _source or some of its fields (or stored fields if not using "_source"). But the point is that the user should be able to specify Fragmenter type (or provide custom implementation of Fragmenter).
>                                                                                     
>                                                                                     
>                                                                                     
> On Thu, Aug 12, 2010 at 8:03 PM, Lukáš Vlček <lukas.vlcek@gmail.com> wrote:
> If I read it correctly then I think it partly overlaps with http://github.com/elasticsearch/elasticsearch/issues/issue/308
>                                                                                             
>                                                                                             
> Lukas
>                                                                                             
>                                                                                             
>                                                                                             
> On Thu, Aug 12, 2010 at 5:53 PM, Shay Banon <shay.banon@elasticsearch.com> wrote:
> Agreed. A bit tricky to implement, but possible. Also, note that this will require loading the full json from the index, and parse it in order to get the relevant parts from it. It won't be returned, but still loaded. So, it might not make sense when you have several big fields in the index, and you want to get fragments for one of them. But does make sense when having one big field.
>                                                                                                     
>                                                                                                     
> Also, if this is implemented, it should also be possible to get specific fields out of the json as a response as well (similar to asking for specific fields in the search request, maybe call them source_fileds).
>                                                                                                     
>                                                                                                     
> Open an issue for this?
>                                                                                                     
>                                                                                                     
> -shay.banon
>                                                                                                     
>                                                                                                     
>                                                                                                     
> On Thu, Aug 12, 2010 at 6:06 PM, Tomislav Poljak <tpoljak@gmail.com> wrote:
> Hi,
> I really like all the features stored json (enabled _source) provides in
> both Java API (used it in indexing/updating) and REST API (used it for
> searching).
>                                                                                                             
> I do however have one possible request for improvement regarding
> documents with large textual fields and overall highlighting.
>                                                                                                             
> When there is a requirement to index/search documents with large textual
> fields (like 'content' with text in Mb, which is not unusual), returning
> a whole json for each result in result set can be impossible (if each
> json document has a few Mb in 'content' returning 30-50 results to a
> client doesn't sound realistic or even possible in acceptable time with
> usual 'Internet' bandwidth).
>                                                                                                             
> But, usually it's acceptable (or even requirement) to display/return
> only highlighting snippets for 30-50 results matches and retrieve whole
> document (json source) only for a single document (when requested by
> exact ID).
>                                                                                                             
> To be able to provide highlighting snippets for large textual
> ('content') field, it needs to be stored.
>                                                                                                             
> Now we are in situation where because textual field is too big (makes
> impassible to return json source for 30-50 results to a client) we need
> to store it twice in index (once as a part of original json source in
> _source and second time as a 'content' field "store" : "yes" for
> highlighting). This make index a lot bigger.
>                                                                                                             
> Also, if there is a requirement to display highlighting for all fields
> (separate highlight snippets for each field where match occurred,
> without mixing fields snippets -> stored _all field can not be used for
> highlighting in such case) then whole document (all fields) needs to be
> stored twice.
>                                                                                                             
> In this case seems only logical to disable _source field (since all
> fields are stored anyway) and when whole document needs to be retrieved
> use (newly added) fields=* feature (I've read similar discussion thread
> which led to this enhancement)
>                                                                                                             
> Here is my question/proposal:
>                                                                                                             
> would it be possible to enable use of json _source field for 'field
> specific' highlighting, where matching snippet needs to be returned
> separately for each field name?
>                                                                                                             
> Maybe to have term_vector for each field, but to somehow 'adjust' or
> recalculate positions_offsets to point to text snippet in _source
> instead of stored field?
>                                                                                                             
> I know this is not a simple requirement, but if 'field specific'
> highlighting could somehow use stored json instead of requiring an
> individual field to be (separately) stored, that would make a great
> use/reuse of stored json _source (and no one would ever think of
> disabling it :)
>                                                                                                             
> Tomislav
> 
> ```

---

<div class="post-metadata">

**Author:** ![Tomislav\_Poljak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomislav_poljak/32/1177_2.png) [@Tomislav\_Poljak](https://discuss.elastic.co/u/Tomislav_Poljak)\
**Post date:** [August 13, 2010, 2:47pm UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213/17 "2010-08-13T14:47:21Z")

</div>

Hi,  
actually this requirement can be even more summarized:

It would be really (really) great if ES could provide 'highlighting' and  
'fields' features from the search API without the need for each field to  
be stored separately (by reusing stored json \_source field).

Do you think this would be possible?

Tomislav

On Fri, 2010-08-13 at 16:30 +0200, Tomislav Poljak wrote:

> Hi,  
> I'm not sure I fully understand what will be implement as a  
> result/conclusion of discussion here, but I think I can define what I  
> would like to be implemented (from my point of view) pretty clearly as:
> 
> It would be great if ES, beside returning whole document source (in json  
> format) in search results, supported returning json type structure with  
> 'matching fields' and/or requested fields. Only fields which are matched  
> by a query or requested would be returned (from \_source json) and this  
> would be possible without storing each field separately. Value in these  
> fields would be either a whole field value (with highlighting applied)  
> or a highlighting snippet (for large textual fields).
> 
> Will something like that be possible?
> 
> Thanks,  
> Tomislav
> 
> On Fri, 2010-08-13 at 12:50 +0300, Shay Banon wrote:
> 
> > No problem, I understand the general idea of the requirement.
> > 
> > -shay.banon
> > 
> > On Fri, Aug 13, 2010 at 11:21 AM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)  
> > wrote:
> > 
> > ```
> > On Fri, Aug 13, 2010 at 10:20 AM, Lukáš Vlček
> > <lukas.vlcek@gmail.com> wrote:
> > Yes, I did not realize that earlier but you are right
> > that I will need to pass query into the highlight
> > section as well.
> > Take the following example:
> >             
> >             
> > I need to display all candidates that match "dude
> > java" query and then I want to allow user to click on
> > individual name and get whole bio highlighted.
> >             
> >             
> > So how I can go about this:
> > First, I can get relevant documents using simple
> > "query_string" query for "dude java". I can now
> > display names of candidates without highlights and
> > highlighted fragments from bio for each name, kind of
> > basic search interface that already works now. But if
> > I wanted to display highlighted name I would get
> > something like "..e <em>Dude</em> Abid..." which is
> > not what I want (sure, I can work with fragment size
> > but that is just workaround and does not fit all
> > situations). So when using that "query_string" query I
> > would like to specify in the highlight section that
> > the person.name should be highlighted with no
> > fragments.
> >             
> >             
> > Second, now, when the user clicks individual name,
> > then I want to get whole bio highlighted.
> > So I need to get specific document (by ID) and have
> > the bio field highlighted (and the name field as well)
> > The example of the query that could be used:
> >             
> > curl -XGET http://localhost:9200/_all/_search -d '
> > { "query" : { "term" : { "person-id" : "1234" } },
> > "highlight" : {
> > "fields" : {
> > "_source" : {
> > "path" : "person.bio,person.name",
> > "fragmenter" : "classpath.to.NullFragmenter",
> > "query" : {
> > "query_string" : { "fields" :
> > ["bio","name"], "query" : "dude java" }
> > }
> > }
> > }
> > }
> > }'
> >             
> >             
> > or I could use fields query:
> >             
> >             
> > curl -XGET http://localhost:9200/_all/_search -d '
> > { "query" : { "term" : { "person-id" : "1234" } },
> > "fields" : ["bio","name"],
> > "highlight" : {
> > "fields" : {
> > "bio" : {
> > "query" : {
> > "query_string" : { "fields" : ["bio"],
> > "query" : "dude java" }
> > }
> > },
> > "name" : {
> > "query" : {
> > "query_string" : { "fields" : ["name"],
> > "query" : "dude java" }
> > }
> > }
> > }
> > }
> > }'
> >             
> >             
> > The later query requires both bio and name to be
> > stored (and this is where it gets back to Tomislav's
> > original point I think).
> > Ugh! I am complicating it way too much... but hope the
> > request is clear now :-)
> >     
> >     
> > Sure I am complicating it too much because in the later query
> > example I forgot the specify NullFragmenter :-)
> >     
> >      
> >             
> >             
> > Regards,
> > Lukas
> >             
> >             
> >             
> > 2010/8/13 Shay Banon <shay.banon@elasticsearch.com>
> >             
> >             
> > ok, so you want to get the whole bio field
> > highlighted, so you would need to pass the
> > query to the get API as well, otherwise, there
> > is no way to highlight it (aside from other
> > things you need, like the option to do no
> > fragmentation and getting the actual data).
> >                     
> >                     
> >                     
> > On Thu, Aug 12, 2010 at 10:34 PM, Lukáš Vlček
> > <lukas.vlcek@gmail.com> wrote:
> > Oh, and one more note, see below:
> >                             
> > On Thu, Aug 12, 2010 at 9:22 PM, Lukáš
> > Vlček <lukas.vlcek@gmail.com> wrote:
> > If I want to display whole bio
> > highlighted then I can either
> > get "_source" and cut bio from
> > it on the client side but in
> > this case I need to tell ES to
> > use highlighting on it first.
> > Or I need to specify in
> > mapping that bio is also
> > stored and use fields
> > query http://www.elasticsearch.com/docs/elasticsearch/rest_api/search/fields/ but again I need to tell ES to highlight it. And in neither case I want only fragments, I want WHOLE content of the field. The first approach is not possible now the later is possible but required bio to be explicitly stored (and it is already stored in _source).
> >                             
> >                             
> > And the later also requires
> > specification of Fragmenter that
> > returns whole body, not fragments,
> > thus my reference to NullFragmenter,
> > which is not implemented in
> > FastVectorHighlighter API (as far as I
> > understand it), it can be found in the
> > older Highlighting API, thus I opened
> > also http://github.com/elasticsearch/elasticsearch/issues/issue/307 
> >                             
> >                             
> > May be it would be better if the
> > NullFragmenter-like functionality is
> > contributed directly into Lucene
> > FastVectorHighlighter API. I was
> > looking at the FVH API today and I
> > think I can try to implement such
> > Fragmenter.
> >                             
> >                             
> >                             
> >                                     
> >                                     
> > Hope this makes it clear.
> > (Sorry if I confused you).
> >                                     
> >                                     
> >                                     
> > Lukas
> >                                     
> > 2010/8/12 Shay Banon
> > <shay.banon@elasticsearch.com>
> > Ahh, I see. So you
> > would still need to
> > provide a query to the
> > GET api in order to do
> > the highlighting,
> > right?
> >                                             
> >                                             
> >                                             
> > On Thu, Aug 12, 2010
> > at 10:08 PM, Lukáš
> > Vlček
> > <lukas.vlcek@gmail.com> wrote:
> > Imagine search
> > app for HR:
> > Candidate
> > catalog (cool
> > name!).
> > The entities
> > stored in the
> > index are as
> > follows:
> > person: { id,
> > name, address,
> > bio }
> > Now I am using
> > just the REST
> > API. Say I
> > search for
> > "Java" and I
> > would like to
> > display list
> > of Names and
> > allow users to
> > click
> > individual
> > name which
> > would display
> > whole bio with
> > Java
> > highlighted in
> > it (here comes
> > the
> > highlighting
> > in the play!).
> > Now I can
> > display bio
> > (just using
> > GET REST API
> > with given
> > document ID
> > but not
> > highlighted.
> > So I was
> > thinking that
> > it would be
> > cool to have
> > this function.
> >                                                     
> >                                                     
> > Lukas
> >                                                     
> > 2010/8/12 Shay
> > Banon
> > <shay.banon@elasticsearch.com>
> >                                                     
> >                                                     
> > So,
> > what
> > you
> > want
> > is to
> > be
> > able
> > to get
> > just
> > the
> > bio
> > field,
> > without the full source, and without the bio field being stored? If so, then the response I gave, where the logic might apply also to get fields using something like "source_field" notion applies here. It does mean that the full source will need to be retrieved and parsed. Not sure how highlighting comes into play here...
> >                                                             
> >                                                             
> > -shay.banon
> >                                                             
> >                                                             
> >                                                             
> > On
> > Thu,
> > Aug
> > 12,
> > 2010
> > at
> > 9:53
> > PM,
> > Lukáš
> > Vlček
> > <lukas.vlcek@gmail.com> wrote:
> > Actually, that ticket has two parts. One is Fragmenter related and the other one is possibility to tell, that I want to highlight some portion of _source data. Imagine I am using only REST API and for example if _source is a person with name, address and bio fields then I would like to tell that I want to highlight just the bio field (and I think the NullFragmenter would be needed for this if I want to display whole content of bio highlighted, not just fragments). The other possibility would be to define mapping for person in such a way that bio would be a stored field, then I could query for stored fields (not pulling the _source field) and tell the I want to apply NullFragmenter to this data while highlighting. But this gets back to the Tomislav's situation, because this would mean that bio is probably stored twice, once as a part of source and then separately as a stored bio field.
> >                                                                     
> >                                                                     
> > Lukas
> >                                                                     
> >                                                                     
> >                                                                     
> > On Thu, Aug 12, 2010 at 8:41 PM, Shay Banon <shay.banon@elasticsearch.com> wrote:
> > Not sure if it overlaps, fragmenter controls how to break the highlighted data, this relates to how to fetch that date to highlight.
> >                                                                             
> >                                                                             
> > -shay.banon
> >                                                                             
> >                                                                             
> >                                                                             
> > On Thu, Aug 12, 2010 at 9:08 PM, Lukáš Vlček <lukas.vlcek@gmail.com> wrote:
> > One of differences is that the 308 issue was meant to return whole content of the _source or some of its fields (or stored fields if not using "_source"). But the point is that the user should be able to specify Fragmenter type (or provide custom implementation of Fragmenter).
> >                                                                                     
> >                                                                                     
> >                                                                                     
> > On Thu, Aug 12, 2010 at 8:03 PM, Lukáš Vlček <lukas.vlcek@gmail.com> wrote:
> > If I read it correctly then I think it partly overlaps with http://github.com/elasticsearch/elasticsearch/issues/issue/308
> >                                                                                             
> >                                                                                             
> > Lukas
> >                                                                                             
> >                                                                                             
> >                                                                                             
> > On Thu, Aug 12, 2010 at 5:53 PM, Shay Banon <shay.banon@elasticsearch.com> wrote:
> > Agreed. A bit tricky to implement, but possible. Also, note that this will require loading the full json from the index, and parse it in order to get the relevant parts from it. It won't be returned, but still loaded. So, it might not make sense when you have several big fields in the index, and you want to get fragments for one of them. But does make sense when having one big field.
> >                                                                                                     
> >                                                                                                     
> > Also, if this is implemented, it should also be possible to get specific fields out of the json as a response as well (similar to asking for specific fields in the search request, maybe call them source_fileds).
> >                                                                                                     
> >                                                                                                     
> > Open an issue for this?
> >                                                                                                     
> >                                                                                                     
> > -shay.banon
> >                                                                                                     
> >                                                                                                     
> >                                                                                                     
> > On Thu, Aug 12, 2010 at 6:06 PM, Tomislav Poljak <tpoljak@gmail.com> wrote:
> > Hi,
> > I really like all the features stored json (enabled _source) provides in
> > both Java API (used it in indexing/updating) and REST API (used it for
> > searching).
> >                                                                                                             
> > I do however have one possible request for improvement regarding
> > documents with large textual fields and overall highlighting.
> >                                                                                                             
> > When there is a requirement to index/search documents with large textual
> > fields (like 'content' with text in Mb, which is not unusual), returning
> > a whole json for each result in result set can be impossible (if each
> > json document has a few Mb in 'content' returning 30-50 results to a
> > client doesn't sound realistic or even possible in acceptable time with
> > usual 'Internet' bandwidth).
> >                                                                                                             
> > But, usually it's acceptable (or even requirement) to display/return
> > only highlighting snippets for 30-50 results matches and retrieve whole
> > document (json source) only for a single document (when requested by
> > exact ID).
> >                                                                                                             
> > To be able to provide highlighting snippets for large textual
> > ('content') field, it needs to be stored.
> >                                                                                                             
> > Now we are in situation where because textual field is too big (makes
> > impassible to return json source for 30-50 results to a client) we need
> > to store it twice in index (once as a part of original json source in
> > _source and second time as a 'content' field "store" : "yes" for
> > highlighting). This make index a lot bigger.
> >                                                                                                             
> > Also, if there is a requirement to display highlighting for all fields
> > (separate highlight snippets for each field where match occurred,
> > without mixing fields snippets -> stored _all field can not be used for
> > highlighting in such case) then whole document (all fields) needs to be
> > stored twice.
> >                                                                                                             
> > In this case seems only logical to disable _source field (since all
> > fields are stored anyway) and when whole document needs to be retrieved
> > use (newly added) fields=* feature (I've read similar discussion thread
> > which led to this enhancement)
> >                                                                                                             
> > Here is my question/proposal:
> >                                                                                                             
> > would it be possible to enable use of json _source field for 'field
> > specific' highlighting, where matching snippet needs to be returned
> > separately for each field name?
> >                                                                                                             
> > Maybe to have term_vector for each field, but to somehow 'adjust' or
> > recalculate positions_offsets to point to text snippet in _source
> > instead of stored field?
> >                                                                                                             
> > I know this is not a simple requirement, but if 'field specific'
> > highlighting could somehow use stored json instead of requiring an
> > individual field to be (separately) stored, that would make a great
> > use/reuse of stored json _source (and no one would ever think of
> > disabling it :)
> >                                                                                                             
> > Tomislav
> > 
> > ```

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 13, 2010, 3:00pm UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213/18 "2010-08-13T15:00:52Z")

</div>

Yea, the discussion took a different turn from the original request. Yes, it  
is possible (with the mentioned downsides of needing to load the full source  
and parsing it on the "fetch" phase within the specific node, nothing that  
can't be solved by adding more replicas though if there are performance  
problems, and you can do that dynamically in upcoming 0.9.1). Can you open a  
feature request for it?

On Fri, Aug 13, 2010 at 5:47 PM, Tomislav Poljak [tpoljak@gmail.com](mailto:tpoljak@gmail.com) wrote:

> Hi,  
> actually this requirement can be even more summarized:
> 
> It would be really (really) great if ES could provide 'highlighting' and  
> 'fields' features from the search API without the need for each field to  
> be stored separately (by reusing stored json \_source field).
> 
> Do you think this would be possible?
> 
> Tomislav
> 
> On Fri, 2010-08-13 at 16:30 +0200, Tomislav Poljak wrote:
> 
> > Hi,  
> > I'm not sure I fully understand what will be implement as a  
> > result/conclusion of discussion here, but I think I can define what I  
> > would like to be implemented (from my point of view) pretty clearly as:
> > 
> > It would be great if ES, beside returning whole document source (in json  
> > format) in search results, supported returning json type structure with  
> > 'matching fields' and/or requested fields. Only fields which are matched  
> > by a query or requested would be returned (from \_source json) and this  
> > would be possible without storing each field separately. Value in these  
> > fields would be either a whole field value (with highlighting applied)  
> > or a highlighting snippet (for large textual fields).
> > 
> > Will something like that be possible?
> > 
> > Thanks,  
> > Tomislav
> > 
> > On Fri, 2010-08-13 at 12:50 +0300, Shay Banon wrote:
> > 
> > > No problem, I understand the general idea of the requirement.
> > > 
> > > -shay.banon
> > > 
> > > On Fri, Aug 13, 2010 at 11:21 AM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)  
> > > wrote:
> > > 
> > > ```
> > > On Fri, Aug 13, 2010 at 10:20 AM, Lukáš Vlček
> > > <lukas.vlcek@gmail.com> wrote:
> > > Yes, I did not realize that earlier but you are right
> > > that I will need to pass query into the highlight
> > > section as well.
> > > Take the following example:
> > > 
> > > I need to display all candidates that match "dude
> > > java" query and then I want to allow user to click on
> > > individual name and get whole bio highlighted.
> > > 
> > > So how I can go about this:
> > > First, I can get relevant documents using simple
> > > "query_string" query for "dude java". I can now
> > > display names of candidates without highlights and
> > > highlighted fragments from bio for each name, kind of
> > > basic search interface that already works now. But if
> > > I wanted to display highlighted name I would get
> > > something like "..e <em>Dude</em> Abid..." which is
> > > not what I want (sure, I can work with fragment size
> > > but that is just workaround and does not fit all
> > > situations). So when using that "query_string" query I
> > > would like to specify in the highlight section that
> > > the person.name should be highlighted with no
> > > fragments.
> > > 
> > > Second, now, when the user clicks individual name,
> > > then I want to get whole bio highlighted.
> > > So I need to get specific document (by ID) and have
> > > the bio field highlighted (and the name field as well)
> > > The example of the query that could be used:
> > > 
> > > curl -XGET http://localhost:9200/_all/_search -d '
> > > { "query" : { "term" : { "person-id" : "1234" } },
> > > "highlight" : {
> > > "fields" : {
> > > "_source" : {
> > > "path" : "person.bio,person.name",
> > > "fragmenter" : "classpath.to.NullFragmenter",
> > > "query" : {
> > > "query_string" : { "fields" :
> > > ["bio","name"], "query" : "dude java" }
> > > }
> > > }
> > > }
> > > }
> > > }'
> > > 
> > > or I could use fields query:
> > > 
> > > curl -XGET http://localhost:9200/_all/_search -d '
> > > { "query" : { "term" : { "person-id" : "1234" } },
> > > "fields" : ["bio","name"],
> > > "highlight" : {
> > > "fields" : {
> > > "bio" : {
> > > "query" : {
> > > "query_string" : { "fields" : ["bio"],
> > > "query" : "dude java" }
> > > }
> > > },
> > > "name" : {
> > > "query" : {
> > > "query_string" : { "fields" : ["name"],
> > > "query" : "dude java" }
> > > }
> > > }
> > > }
> > > }
> > > }'
> > > 
> > > The later query requires both bio and name to be
> > > stored (and this is where it gets back to Tomislav's
> > > original point I think).
> > > Ugh! I am complicating it way too much... but hope the
> > > request is clear now :-)
> > > 
> > > Sure I am complicating it too much because in the later query
> > > example I forgot the specify NullFragmenter :-)
> > > 
> > > Regards,
> > > Lukas
> > > 
> > > 2010/8/13 Shay Banon <shay.banon@elasticsearch.com>
> > > 
> > > ok, so you want to get the whole bio field
> > > highlighted, so you would need to pass the
> > > query to the get API as well, otherwise, there
> > > is no way to highlight it (aside from other
> > > things you need, like the option to do no
> > > fragmentation and getting the actual data).
> > > 
> > > On Thu, Aug 12, 2010 at 10:34 PM, Lukáš Vlček
> > > <lukas.vlcek@gmail.com> wrote:
> > > Oh, and one more note, see below:
> > > 
> > > On Thu, Aug 12, 2010 at 9:22 PM, Lukáš
> > > Vlček <lukas.vlcek@gmail.com> wrote:
> > > If I want to display whole bio
> > > highlighted then I can either
> > > get "_source" and cut bio from
> > > it on the client side but in
> > > this case I need to tell ES to
> > > use highlighting on it first.
> > > Or I need to specify in
> > > mapping that bio is also
> > > stored and use fields
> > > query
> > > 
> > > ```
> 
> [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/search/fields/but](http://www.elasticsearch.com/docs/elasticsearch/rest_api/search/fields/but) again I need to tell ES to highlight it. And in neither case I want only  
> fragments, I want WHOLE content of the field. The first approach is not  
> possible now the later is possible but required bio to be explicitly stored  
> (and it is already stored in \_source).
> 
> > > ```
> > > And the later also requires
> > > specification of Fragmenter that
> > > returns whole body, not fragments,
> > > thus my reference to NullFragmenter,
> > > which is not implemented in
> > > FastVectorHighlighter API (as far as I
> > > understand it), it can be found in the
> > > older Highlighting API, thus I opened
> > > also
> > > 
> > > ```
> 
> [Highlighting API: Add support for custom FragmentsBuilder · Issue #307 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/307)
> 
> > > ```
> > > May be it would be better if the
> > > NullFragmenter-like functionality is
> > > contributed directly into Lucene
> > > FastVectorHighlighter API. I was
> > > looking at the FVH API today and I
> > > think I can try to implement such
> > > Fragmenter.
> > > 
> > > Hope this makes it clear.
> > > (Sorry if I confused you).
> > > 
> > > Lukas
> > > 
> > > 2010/8/12 Shay Banon
> > > <shay.banon@elasticsearch.com>
> > > Ahh, I see. So you
> > > would still need to
> > > provide a query to the
> > > GET api in order to do
> > > the highlighting,
> > > right?
> > > 
> > > On Thu, Aug 12, 2010
> > > at 10:08 PM, Lukáš
> > > Vlček
> > > <lukas.vlcek@gmail.com>
> > > 
> > > ```
> 
> wrote:
> 
> > > ```
> > > Imagine search
> > > app for HR:
> > > Candidate
> > > catalog (cool
> > > name!).
> > > The entities
> > > stored in the
> > > index are as
> > > follows:
> > > person: { id,
> > > name, address,
> > > bio }
> > > Now I am using
> > > just the REST
> > > API. Say I
> > > search for
> > > "Java" and I
> > > would like to
> > > display list
> > > of Names and
> > > allow users to
> > > click
> > > individual
> > > name which
> > > would display
> > > whole bio with
> > > Java
> > > highlighted in
> > > it (here comes
> > > the
> > > highlighting
> > > in the play!).
> > > Now I can
> > > display bio
> > > (just using
> > > GET REST API
> > > with given
> > > document ID
> > > but not
> > > highlighted.
> > > So I was
> > > thinking that
> > > it would be
> > > cool to have
> > > this function.
> > > 
> > > Lukas
> > > 
> > > 2010/8/12 Shay
> > > Banon
> > > <
> > > 
> > > ```
> 
> [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\>
> 
> > > ```
> > > So,
> > > what
> > > you
> > > want
> > > is to
> > > be
> > > able
> > > to get
> > > just
> > > the
> > > bio
> > > field,
> > > without
> > > 
> > > ```
> 
> the full source, and without the bio field being stored? If so, then the  
> response I gave, where the logic might apply also to get fields using  
> something like "source\_field" notion applies here. It does mean that the  
> full source will need to be retrieved and parsed. Not sure how highlighting  
> comes into play here...
> 
> > >
> 
> -shay.banon
> 
> > > ```
> > > On
> > > Thu,
> > > Aug
> > > 12,
> > > 2010
> > > at
> > > 9:53
> > > PM,
> > > Lukáš
> > > Vlček
> > > <
> > > 
> > > ```
> 
> [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)\> wrote:
> 
> > >
> 
> Actually, that ticket has two parts. One is Fragmenter related and the other  
> one is possibility to tell, that I want to highlight some portion of \_source  
> data. Imagine I am using only REST API and for example if \_source is a  
> person with name, address and bio fields then I would like to tell that I  
> want to highlight just the bio field (and I think the NullFragmenter would  
> be needed for this if I want to display whole content of bio highlighted,  
> not just fragments). The other possibility would be to define mapping for  
> person in such a way that bio would be a stored field, then I could query  
> for stored fields (not pulling the \_source field) and tell the I want to  
> apply NullFragmenter to this data while highlighting. But this gets back to  
> the Tomislav's situation, because this would mean that bio is probably  
> stored twice, once as a part of source and then separately as a stored bio  
> field.
> 
> > >
> 
> Lukas
> 
> > >
> 
> On Thu, Aug 12, 2010 at 8:41 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)  
> wrote:
> 
> > >
> 
> ```
> Not sure if it overlaps, fragmenter controls how to break the
> 
> ```
> 
> highlighted data, this relates to how to fetch that date to highlight.
> 
> > >
> 
> ```
> -shay.banon
> 
> ```
> 
> > >
> 
> ```
> On Thu, Aug 12, 2010 at 9:08 PM, Lukáš Vlček <
> 
> ```
> 
> [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)\> wrote:
> 
> > >
> 
> ```
> One of differences is that the 308 issue was meant to return
> 
> ```
> 
> whole content of the \_source or some of its fields (or stored fields if not  
> using "\_source"). But the point is that the user should be able to specify  
> Fragmenter type (or provide custom implementation of Fragmenter).
> 
> > >
> 
> ```
> On Thu, Aug 12, 2010 at 8:03 PM, Lukáš Vlček <
> 
> ```
> 
> [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)\> wrote:
> 
> > >
> 
> ```
> If I read it correctly then I think it partly
> 
> ```
> 
> overlaps with  
> [Highlighting API: Apply highlighting to REST GET operation · Issue #308 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/308)
> 
> > >
> 
> ```
> Lukas
> 
> ```
> 
> > >
> 
> ```
> On Thu, Aug 12, 2010 at 5:53 PM, Shay Banon <
> 
> ```
> 
> [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> 
> > >
> 
> ```
> Agreed. A bit tricky to implement, but
> 
> ```
> 
> possible. Also, note that this will require loading the full json from the  
> index, and parse it in order to get the relevant parts from it. It won't be  
> returned, but still loaded. So, it might not make sense when you have  
> several big fields in the index, and you want to get fragments for one of  
> them. But does make sense when having one big field.
> 
> > >
> 
> ```
> Also, if this is implemented, it should also
> 
> ```
> 
> be possible to get specific fields out of the json as a response as well  
> (similar to asking for specific fields in the search request, maybe call  
> them source\_fileds).
> 
> > >
> 
> ```
> Open an issue for this?
> 
> ```
> 
> > >
> 
> ```
> -shay.banon
> 
> ```
> 
> > >
> 
> ```
> On Thu, Aug 12, 2010 at 6:06 PM, Tomislav
> 
> ```
> 
> Poljak [tpoljak@gmail.com](mailto:tpoljak@gmail.com) wrote:
> 
> > >
> 
> ```
> Hi,
> 
> ```
> 
> > >
> 
> ```
> I really like all the features
> 
> ```
> 
> stored json (enabled \_source) provides in
> 
> > >
> 
> ```
> both Java API (used it in
> 
> ```
> 
> indexing/updating) and REST API (used it for
> 
> > >
> 
> ```
> searching).
> 
> ```
> 
> > >
> 
> ```
> I do however have one possible
> 
> ```
> 
> request for improvement regarding
> 
> > >
> 
> ```
> documents with large textual fields
> 
> ```
> 
> and overall highlighting.
> 
> > >
> 
> ```
> When there is a requirement to
> 
> ```
> 
> index/search documents with large textual
> 
> > >
> 
> ```
> fields (like 'content' with text in
> 
> ```
> 
> Mb, which is not unusual), returning
> 
> > >
> 
> ```
> a whole json for each result in
> 
> ```
> 
> result set can be impossible (if each
> 
> > >
> 
> ```
> json document has a few Mb in
> 
> ```
> 
> 'content' returning 30-50 results to a
> 
> > >
> 
> ```
> client doesn't sound realistic or
> 
> ```
> 
> even possible in acceptable time with
> 
> > >
> 
> ```
> usual 'Internet' bandwidth).
> 
> ```
> 
> > >
> 
> ```
> But, usually it's acceptable (or
> 
> ```
> 
> even requirement) to display/return
> 
> > >
> 
> ```
> only highlighting snippets for 30-50
> 
> ```
> 
> results matches and retrieve whole
> 
> > >
> 
> ```
> document (json source) only for a
> 
> ```
> 
> single document (when requested by
> 
> > >
> 
> ```
> exact ID).
> 
> ```
> 
> > >
> 
> ```
> To be able to provide highlighting
> 
> ```
> 
> snippets for large textual
> 
> > >
> 
> ```
> ('content') field, it needs to be
> 
> ```
> 
> stored.
> 
> > >
> 
> ```
> Now we are in situation where
> 
> ```
> 
> because textual field is too big (makes
> 
> > >
> 
> ```
> impassible to return json source for
> 
> ```
> 
> 30-50 results to a client) we need
> 
> > >
> 
> ```
> to store it twice in index (once as
> 
> ```
> 
> a part of original json source in
> 
> > >
> 
> ```
> _source and second time as a
> 
> ```
> 
> 'content' field "store" : "yes" for
> 
> > >
> 
> ```
> highlighting). This make index a lot
> 
> ```
> 
> bigger.
> 
> > >
> 
> ```
> Also, if there is a requirement to
> 
> ```
> 
> display highlighting for all fields
> 
> > >
> 
> ```
> (separate highlight snippets for
> 
> ```
> 
> each field where match occurred,
> 
> > >
> 
> ```
> without mixing fields snippets ->
> 
> ```
> 
> stored \_all field can not be used for
> 
> > >
> 
> ```
> highlighting in such case) then
> 
> ```
> 
> whole document (all fields) needs to be
> 
> > >
> 
> ```
> stored twice.
> 
> ```
> 
> > >
> 
> ```
> In this case seems only logical to
> 
> ```
> 
> disable \_source field (since all
> 
> > >
> 
> ```
> fields are stored anyway) and when
> 
> ```
> 
> whole document needs to be retrieved
> 
> > >
> 
> ```
> use (newly added) fields=* feature
> 
> ```
> 
> (I've read similar discussion thread
> 
> > >
> 
> ```
> which led to this enhancement)
> 
> ```
> 
> > >
> 
> ```
> Here is my question/proposal:
> 
> ```
> 
> > >
> 
> ```
> would it be possible to enable use
> 
> ```
> 
> of json \_source field for 'field
> 
> > >
> 
> ```
> specific' highlighting, where
> 
> ```
> 
> matching snippet needs to be returned
> 
> > >
> 
> ```
> separately for each field name?
> 
> ```
> 
> > >
> 
> ```
> Maybe to have term_vector for each
> 
> ```
> 
> field, but to somehow 'adjust' or
> 
> > >
> 
> ```
> recalculate positions_offsets to
> 
> ```
> 
> point to text snippet in \_source
> 
> > >
> 
> ```
> instead of stored field?
> 
> ```
> 
> > >
> 
> ```
> I know this is not a simple
> 
> ```
> 
> requirement, but if 'field specific'
> 
> > >
> 
> ```
> highlighting could somehow use
> 
> ```
> 
> stored json instead of requiring an
> 
> > >
> 
> ```
> individual field to be (separately)
> 
> ```
> 
> stored, that would make a great
> 
> > >
> 
> ```
> use/reuse of stored json _source
> 
> ```
> 
> (and no one would ever think of
> 
> > >
> 
> ```
> disabling it :)
> 
> ```
> 
> > >
> 
> ```
> Tomislav
> 
> ```
> 
> > >

---

<div class="post-metadata">

**Author:** ![Tomislav\_Poljak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomislav_poljak/32/1177_2.png) [@Tomislav\_Poljak](https://discuss.elastic.co/u/Tomislav_Poljak)\
**Post date:** [August 13, 2010, 3:20pm UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213/19 "2010-08-13T15:20:00Z")

</div>

Hi,  
I've opened issue 319 for it, here

['Highlighting' and 'fields' without stored fields · Issue #319 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/319)

Hope this is fine,

```
 Tomislav

```

On Fri, 2010-08-13 at 18:00 +0300, Shay Banon wrote:

> Yea, the discussion took a different turn from the original request.  
> Yes, it is possible (with the mentioned downsides of needing to load  
> the full source and parsing it on the "fetch" phase within the  
> specific node, nothing that can't be solved by adding more replicas  
> though if there are performance problems, and you can do that  
> dynamically in upcoming 0.9.1). Can you open a feature request for it?
> 
> On Fri, Aug 13, 2010 at 5:47 PM, Tomislav Poljak [tpoljak@gmail.com](mailto:tpoljak@gmail.com)  
> wrote:  
> Hi,  
> actually this requirement can be even more summarized:
> 
> ```
> It would be really (really) great if ES could provide
> 'highlighting' and
> 'fields' features from the search API without the need for
> each field to
> be stored separately (by reusing stored json _source field).
>     
> Do you think this would be possible?
>     
> Tomislav
>     
>     
>     
> On Fri, 2010-08-13 at 16:30 +0200, Tomislav Poljak wrote:
> > Hi,
> > I'm not sure I fully understand what will be implement as a
> > result/conclusion of discussion here, but I think I can
> define what I
> > would like to be implemented (from my point of view) pretty
> clearly as:
> >
> > It would be great if ES, beside returning whole document
> source (in json
> > format) in search results, supported returning json type
> structure with
> > 'matching fields' and/or requested fields. Only fields which
> are matched
> > by a query or requested would be returned (from _source
> json) and this
> > would be possible without storing each field separately.
> Value in these
> > fields would be either a whole field value (with
> highlighting applied)
> > or a highlighting snippet (for large textual fields).
> >
> > Will something like that be possible?
> >
> > Thanks,
> > Tomislav
> >
> >
> > On Fri, 2010-08-13 at 12:50 +0300, Shay Banon wrote:
> > > No problem, I understand the general idea of the
> requirement.
> > >
> > >
> > > -shay.banon
> > >
> > > On Fri, Aug 13, 2010 at 11:21 AM, Lukáš Vlček
> <lukas.vlcek@gmail.com>
> > > wrote:
> > >
> > >
> > >
> > > On Fri, Aug 13, 2010 at 10:20 AM, Lukáš Vlček
> > > <lukas.vlcek@gmail.com> wrote:
> > > Yes, I did not realize that earlier but
> you are right
> > > that I will need to pass query into the
> highlight
> > > section as well.
> > > Take the following example:
> > >
> > >
> > > I need to display all candidates that
> match "dude
> > > java" query and then I want to allow user
> to click on
> > > individual name and get whole bio
> highlighted.
> > >
> > >
> > > So how I can go about this:
> > > First, I can get relevant documents using
> simple
> > > "query_string" query for "dude java". I
> can now
> > > display names of candidates without
> highlights and
> > > highlighted fragments from bio for each
> name, kind of
> > > basic search interface that already works
> now. But if
> > > I wanted to display highlighted name I
> would get
> > > something like "..e <em>Dude</em> Abid..."
> which is
> > > not what I want (sure, I can work with
> fragment size
> > > but that is just workaround and does not
> fit all
> > > situations). So when using that
> "query_string" query I
> > > would like to specify in the highlight
> section that
> > > the person.name should be highlighted with
> no
> > > fragments.
> > >
> > >
> > > Second, now, when the user clicks
> individual name,
> > > then I want to get whole bio highlighted.
> > > So I need to get specific document (by ID)
> and have
> > > the bio field highlighted (and the name
> field as well)
> > > The example of the query that could be
> used:
> > >
> > > curl -XGET
> http://localhost:9200/_all/_search -d '
> > > { "query" : { "term" : { "person-id" :
> "1234" } },
> > > "highlight" : {
> > > "fields" : {
> > > "_source" : {
> > > "path" : "person.bio,person.name",
> > > "fragmenter" :
> "classpath.to.NullFragmenter",
> > > "query" : {
> > > "query_string" : { "fields" :
> > > ["bio","name"], "query" : "dude java" }
> > > }
> > > }
> > > }
> > > }
> > > }'
> > >
> > >
> > > or I could use fields query:
> > >
> > >
> > > curl -XGET
> http://localhost:9200/_all/_search -d '
> > > { "query" : { "term" : { "person-id" :
> "1234" } },
> > > "fields" : ["bio","name"],
> > > "highlight" : {
> > > "fields" : {
> > > "bio" : {
> > > "query" : {
> > > "query_string" : { "fields" :
> ["bio"],
> > > "query" : "dude java" }
> > > }
> > > },
> > > "name" : {
> > > "query" : {
> > > "query_string" : { "fields" :
> ["name"],
> > > "query" : "dude java" }
> > > }
> > > }
> > > }
> > > }
> > > }'
> > >
> > >
> > > The later query requires both bio and name
> to be
> > > stored (and this is where it gets back to
> Tomislav's
> > > original point I think).
> > > Ugh! I am complicating it way too much...
> but hope the
> > > request is clear now :-)
> > >
> > >
> > > Sure I am complicating it too much because in the
> later query
> > > example I forgot the specify NullFragmenter :-)
> > >
> > >
> > >
> > >
> > > Regards,
> > > Lukas
> > >
> > >
> > >
> > > 2010/8/13 Shay Banon
> <shay.banon@elasticsearch.com>
> > >
> > >
> > > ok, so you want to get the whole
> bio field
> > > highlighted, so you would need to
> pass the
> > > query to the get API as well,
> otherwise, there
> > > is no way to highlight it (aside
> from other
> > > things you need, like the option
> to do no
> > > fragmentation and getting the
> actual data).
> > >
> > >
> > >
> > > On Thu, Aug 12, 2010 at 10:34 PM,
> Lukáš Vlček
> > > <lukas.vlcek@gmail.com> wrote:
> > > Oh, and one more note, see
> below:
> > >
> > > On Thu, Aug 12, 2010 at
> 9:22 PM, Lukáš
> > > Vlček
> <lukas.vlcek@gmail.com> wrote:
> > > If I want to
> display whole bio
> > > highlighted then I
> can either
> > > get "_source" and
> cut bio from
> > > it on the client
> side but in
> > > this case I need
> to tell ES to
> > > use highlighting
> on it first.
> > > Or I need to
> specify in
> > > mapping that bio
> is also
> > > stored and use
> fields
> > > query
> http://www.elasticsearch.com/docs/elasticsearch/rest_api/search/fields/ but again I need to tell ES to highlight it. And in neither case I want only fragments, I want WHOLE content of the field. The first approach is not possible now the later is possible but required bio to be explicitly stored (and it is already stored in _source).
> > >
> > >
> > > And the later also
> requires
> > > specification of
> Fragmenter that
> > > returns whole body, not
> fragments,
> > > thus my reference to
> NullFragmenter,
> > > which is not implemented
> in
> > > FastVectorHighlighter API
> (as far as I
> > > understand it), it can be
> found in the
> > > older Highlighting API,
> thus I opened
> > > also
> http://github.com/elasticsearch/elasticsearch/issues/issue/307
> > >
> > >
> > > May be it would be better
> if the
> > > NullFragmenter-like
> functionality is
> > > contributed directly into
> Lucene
> > > FastVectorHighlighter API.
> I was
> > > looking at the FVH API
> today and I
> > > think I can try to
> implement such
> > > Fragmenter.
> > >
> > >
> > >
> > >
> > >
> > > Hope this makes it
> clear.
> > > (Sorry if I
> confused you).
> > >
> > >
> > >
> > > Lukas
> > >
> > > 2010/8/12 Shay
> Banon
> > >
> <shay.banon@elasticsearch.com>
> > > Ahh, I
> see. So you
> > > would
> still need to
> > > provide a
> query to the
> > > GET api in
> order to do
> > > the
> highlighting,
> > > right?
> > >
> > >
> > >
> > > On Thu,
> Aug 12, 2010
> > > at 10:08
> PM, Lukáš
> > > Vlček
> > >
> <lukas.vlcek@gmail.com> wrote:
> > >
> Imagine search
> > >
> app for HR:
> > >
> Candidate
> > >
> catalog (cool
> > >
> name!).
> > >
> The entities
> > >
> stored in the
> > >
> index are as
> > >
> follows:
> > >
> person: { id,
> > >
> name, address,
> > >
> bio }
> > >
> Now I am using
> > >
> just the REST
> > >
> API. Say I
> > >
> search for
> > >
> "Java" and I
> > >
> would like to
> > >
> display list
> > > of
> Names and
> > >
> allow users to
> > >
> click
> > >
> individual
> > >
> name which
> > >
> would display
> > >
> whole bio with
> > >
> Java
> > >
> highlighted in
> > > it
> (here comes
> > >
> the
> > >
> highlighting
> > > in
> the play!).
> > >
> Now I can
> > >
> display bio
> > >
> (just using
> > >
> GET REST API
> > >
> with given
> > >
> document ID
> > >
> but not
> > >
> highlighted.
> > > So
> I was
> > >
> thinking that
> > > it
> would be
> > >
> cool to have
> > >
> this function.
> > >
> > >
> > >
> Lukas
> > >
> > >
> 2010/8/12 Shay
> > >
> Banon
> > >
> <shay.banon@elasticsearch.com>
> > >
> > >
> > >
> So,
> > >
> what
> > >
> you
> > >
> want
> > >
> is to
> > >
> be
> > >
> able
> > >
> to get
> > >
> just
> > >
> the
> > >
> bio
> > >
> field,
> > >
> without the full source, and without the bio field being
> stored? If so, then the response I gave, where the logic might
> apply also to get fields using something like "source_field"
> notion applies here. It does mean that the full source will
> need to be retrieved and parsed. Not sure how highlighting
> comes into play here...
> > >
> > >
> > >
> -shay.banon
> > >
> > >
> > >
> > >
> On
> > >
> Thu,
> > >
> Aug
> > >
> 12,
> > >
> 2010
> > >
> at
> > >
> 9:53
> > >
> PM,
> > >
> Lukáš
> > >
> Vlček
> > >
> <lukas.vlcek@gmail.com> wrote:
> > >
> Actually, that ticket has two parts. One is Fragmenter related
> and the other one is possibility to tell, that I want to
> highlight some portion of _source data. Imagine I am using
> only REST API and for example if _source is a person with
> name, address and bio fields then I would like to tell that I
> want to highlight just the bio field (and I think the
> NullFragmenter would be needed for this if I want to display
> whole content of bio highlighted, not just fragments). The
> other possibility would be to define mapping for person in
> such a way that bio would be a stored field, then I could
> query for stored fields (not pulling the _source field) and
> tell the I want to apply NullFragmenter to this data while
> highlighting. But this gets back to the Tomislav's situation,
> because this would mean that bio is probably stored twice,
> once as a part of source and then separately as a stored bio
> field.
> > >
> > >
> > >
> Lukas
> > >
> > >
> > >
> > >
> On Thu, Aug 12, 2010 at 8:41 PM, Shay Banon
> <shay.banon@elasticsearch.com> wrote:
> > >
> Not sure if it overlaps, fragmenter controls how to break the
> highlighted data, this relates to how to fetch that date to
> highlight.
> > >
> > >
> > >
> -shay.banon
> > >
> > >
> > >
> > >
> On Thu, Aug 12, 2010 at 9:08 PM, Lukáš Vlček
> <lukas.vlcek@gmail.com> wrote:
> > >
> One of differences is that the 308 issue was meant to return
> whole content of the _source or some of its fields (or stored
> fields if not using "_source"). But the point is that the user
> should be able to specify Fragmenter type (or provide custom
> implementation of Fragmenter).
> > >
> > >
> > >
> > >
> On Thu, Aug 12, 2010 at 8:03 PM, Lukáš Vlček
> <lukas.vlcek@gmail.com> wrote:
> > >
> If I read it correctly then I think it partly overlaps with
> http://github.com/elasticsearch/elasticsearch/issues/issue/308
> > >
> > >
> > >
> Lukas
> > >
> > >
> > >
> > >
> On Thu, Aug 12, 2010 at 5:53 PM, Shay Banon
> <shay.banon@elasticsearch.com> wrote:
> > >
> Agreed. A bit tricky to implement, but possible. Also, note
> that this will require loading the full json from the index,
> and parse it in order to get the relevant parts from it. It
> won't be returned, but still loaded. So, it might not make
> sense when you have several big fields in the index, and you
> want to get fragments for one of them. But does make sense
> when having one big field.
> > >
> > >
> > >
> Also, if this is implemented, it should also be possible to
> get specific fields out of the json as a response as well
> (similar to asking for specific fields in the search request,
> maybe call them source_fileds).
> > >
> > >
> > >
> Open an issue for this?
> > >
> > >
> > >
> -shay.banon
> > >
> > >
> > >
> > >
> On Thu, Aug 12, 2010 at 6:06 PM, Tomislav Poljak
> <tpoljak@gmail.com> wrote:
> > >
> Hi,
> > >
> I really like all the features stored json (enabled _source)
> provides in
> > >
> both Java API (used it in indexing/updating) and REST API
> (used it for
> > >
> searching).
> > >
> > >
> I do however have one possible request for improvement
> regarding
> > >
> documents with large textual fields and overall highlighting.
> > >
> > >
> When there is a requirement to index/search documents with
> large textual
> > >
> fields (like 'content' with text in Mb, which is not unusual),
> returning
> > >
> a whole json for each result in result set can be impossible
> (if each
> > >
> json document has a few Mb in 'content' returning 30-50
> results to a
> > >
> client doesn't sound realistic or even possible in acceptable
> time with
> > >
> usual 'Internet' bandwidth).
> > >
> > >
> But, usually it's acceptable (or even requirement) to
> display/return
> > >
> only highlighting snippets for 30-50 results matches and
> retrieve whole
> > >
> document (json source) only for a single document (when
> requested by
> > >
> exact ID).
> > >
> > >
> To be able to provide highlighting snippets for large textual
> > >
> ('content') field, it needs to be stored.
> > >
> > >
> Now we are in situation where because textual field is too big
> (makes
> > >
> impassible to return json source for 30-50 results to a
> client) we need
> > >
> to store it twice in index (once as a part of original json
> source in
> > >
> _source and second time as a 'content' field "store" : "yes"
> for
> > >
> highlighting). This make index a lot bigger.
> > >
> > >
> Also, if there is a requirement to display highlighting for
> all fields
> > >
> (separate highlight snippets for each field where match
> occurred,
> > >
> without mixing fields snippets -> stored _all field can not be
> used for
> > >
> highlighting in such case) then whole document (all fields)
> needs to be
> > >
> stored twice.
> > >
> > >
> In this case seems only logical to disable _source field
> (since all
> > >
> fields are stored anyway) and when whole document needs to be
> retrieved
> > >
> use (newly added) fields=* feature (I've read similar
> discussion thread
> > >
> which led to this enhancement)
> > >
> > >
> Here is my question/proposal:
> > >
> > >
> would it be possible to enable use of json _source field for
> 'field
> > >
> specific' highlighting, where matching snippet needs to be
> returned
> > >
> separately for each field name?
> > >
> > >
> Maybe to have term_vector for each field, but to somehow
> 'adjust' or
> > >
> recalculate positions_offsets to point to text snippet in
> _source
> > >
> instead of stored field?
> > >
> > >
> I know this is not a simple requirement, but if 'field
> specific'
> > >
> highlighting could somehow use stored json instead of
> requiring an
> > >
> individual field to be (separately) stored, that would make a
> great
> > >
> use/reuse of stored json _source (and no one would ever think
> of
> > >
> disabling it :)
> > >
> > >
> Tomislav
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> 
> ```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:20am UTC](https://discuss.elastic.co/t/large-stored-fields-json-source-and-highlighting/3213/20 "2017-07-06T04:20:47Z")

</div>