When to use fields and when to use source filtering

I read about two ways to filter the fields returned by elasticsearch. fields
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-fields.html
And source filtering
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-source-filtering.html.
when to use which?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0a8593ca-d17f-4f6d-b3b8-b5ee10196892%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Fields are used to pull data from stored fields whereas source filtering is
targeting _source. At the moment both fallback on each other, so the
differences is in the order of precedence. I believe I've heard there're
plans to deprecate fields completely, wonder if someone from ES could
confirm?

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Mon, Dec 22, 2014 at 2:28 PM, Shelef shlaflaf@gmail.com wrote:

I read about two ways to filter the fields returned by elasticsearch.
fields
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-fields.html
And source filtering
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-source-filtering.html.
when to use which?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0a8593ca-d17f-4f6d-b3b8-b5ee10196892%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0a8593ca-d17f-4f6d-b3b8-b5ee10196892%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtVDGbnuBdGgxHavr3O9ToO%2BC_5Tg%3DE8tntw_5yW%3Djm%3Dg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

General rule:

  • Use source filtering unless you can't. Source filtering works if the
    field is in the document you indexed. Fields is required if you want to
    load a stored field. You only need to store fields if they are synthetic
    like from word count or from transform.

Advanced thing I've never used:

  • Its possible to get some performance gains by setting a small field to
    stored and fetching it using fields rather than source filtering if the
    document is large and the field is small and you only want the small field.

Nik
On Dec 22, 2014 7:29 AM, "Shelef" shlaflaf@gmail.com wrote:

I read about two ways to filter the fields returned by elasticsearch.
fields
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-fields.html
And source filtering
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-source-filtering.html.
when to use which?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0a8593ca-d17f-4f6d-b3b8-b5ee10196892%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0a8593ca-d17f-4f6d-b3b8-b5ee10196892%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0yMv9%3DdjpAOUQ5P%3D0dB1GN8DATghhk2zZgsk%3De14Y8jA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Does source fallback? I remember trying and getting nothing.
On Dec 22, 2014 7:33 AM, "Itamar Syn-Hershko" itamar@code972.com wrote:

Fields are used to pull data from stored fields whereas source filtering
is targeting _source. At the moment both fallback on each other, so the
differences is in the order of precedence. I believe I've heard there're
plans to deprecate fields completely, wonder if someone from ES could
confirm?

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Mon, Dec 22, 2014 at 2:28 PM, Shelef shlaflaf@gmail.com wrote:

I read about two ways to filter the fields returned by elasticsearch.
fields
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-fields.html
And source filtering
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-source-filtering.html.
when to use which?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0a8593ca-d17f-4f6d-b3b8-b5ee10196892%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0a8593ca-d17f-4f6d-b3b8-b5ee10196892%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtVDGbnuBdGgxHavr3O9ToO%2BC_5Tg%3DE8tntw_5yW%3Djm%3Dg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtVDGbnuBdGgxHavr3O9ToO%2BC_5Tg%3DE8tntw_5yW%3Djm%3Dg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3Q1uf86aPqy4uo9cbpOerPKDONViCwPog6AT7nwiEHJg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

From the docs:

By default operations return the contents of the _source field unless you
have used the fieldsparameter or if the _source field is disabled.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-source-filtering.html

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Mon, Dec 22, 2014 at 2:46 PM, Nikolas Everett nik9000@gmail.com wrote:

Does source fallback? I remember trying and getting nothing.
On Dec 22, 2014 7:33 AM, "Itamar Syn-Hershko" itamar@code972.com wrote:

Fields are used to pull data from stored fields whereas source filtering
is targeting _source. At the moment both fallback on each other, so the
differences is in the order of precedence. I believe I've heard there're
plans to deprecate fields completely, wonder if someone from ES could
confirm?

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Mon, Dec 22, 2014 at 2:28 PM, Shelef shlaflaf@gmail.com wrote:

I read about two ways to filter the fields returned by elasticsearch.
fields
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-fields.html
And source filtering
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-source-filtering.html.
when to use which?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0a8593ca-d17f-4f6d-b3b8-b5ee10196892%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0a8593ca-d17f-4f6d-b3b8-b5ee10196892%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtVDGbnuBdGgxHavr3O9ToO%2BC_5Tg%3DE8tntw_5yW%3Djm%3Dg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtVDGbnuBdGgxHavr3O9ToO%2BC_5Tg%3DE8tntw_5yW%3Djm%3Dg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3Q1uf86aPqy4uo9cbpOerPKDONViCwPog6AT7nwiEHJg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3Q1uf86aPqy4uo9cbpOerPKDONViCwPog6AT7nwiEHJg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZscB0wvnAGN0%2B6GOF%2BCSV7u-ziH4ybBfpj%3DUa-c7NNUzg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

On Mon, Dec 22, 2014 at 1:33 PM, Itamar Syn-Hershko itamar@code972.com
wrote:

Fields are used to pull data from stored fields whereas source filtering
is targeting _source. At the moment both fallback on each other, so the
differences is in the order of precedence. I believe I've heard there're
plans to deprecate fields completely, wonder if someone from ES could
confirm?

I don't think we can deprecate fields because of synthetic fields
(otherwise you could not eg. highlight them). But it's true that we'd like
to make it unnecessary for you to store a field if you already have it in
the _source. Today _source filtering has the drawback that you might need
to parse a lot of json to get a single value, this is a reason why users
sometimes prefer storing some fields on their own so that they can be
retrieved without parsing the _source. We are thinking about ways to
mitigate this issue, see eg.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j54F4_uQTS_K39r3g_tAuY%3DHfMGhh%2BFeMwHDo00Zytj5Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.