# Null values for optional fields **URL:** \ **Category:** Elasticsearch\ **Created:** [September 27, 2011, 2:59am UTC](https://discuss.elastic.co/t/null-values-for-optional-fields/5458 "2011-09-27T02:59:13Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![Alex\_2](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@Alex\_2](https://discuss.elastic.co/u/Alex_2)\ **Post date:** [September 27, 2011, 2:59am UTC](https://discuss.elastic.co/t/null-values-for-optional-fields/5458/1 "2011-09-27T02:59:13Z")
Hi Shay I found a problem that arises when using this line: getResponse = client.prepareGet(INDEX\_NAME, TYPE, documentId) .setFields("\_source.field1", "\_source.field2", "\_source.optional\_field") .setOperationThreaded(false) .execute() .actionGet() If the optional\_field is missing, the response's fields sometime contains no "\_source.field1" (as expected), but other times it contains a "\_source.field1", whose values are null. Why is the behavior not consistent? Am I missing anything? Thank you! Cheers Alex ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 27, 2011, 9:03pm UTC](https://discuss.elastic.co/t/null-values-for-optional-fields/5458/2 "2011-09-27T21:03:50Z")
Not sure I understand, if optional\_field is missing, then you expect the response _not_ to include field1? Btw, in master, there is much better support for extracting fields from source, you no longer (and should not) use the \_source prefix when it comes out (if you do, it will still use the scripting hack to fetch it). On Tue, Sep 27, 2011 at 5:59 AM, Alex [alessandro.usseglioviretta@gmail.com](mailto:alessandro.usseglioviretta@gmail.com)wrote: > Hi Shay > > I found a problem that arises when using this line: > > getResponse = client.prepareGet(INDEX\_NAME, TYPE, documentId) > .setFields("\_source.field1", "\_source.field2", > "\_source.optional\_field") > .setOperationThreaded(false) > .execute() > .actionGet() > > If the optional\_field is missing, the response's fields sometime contains > no "\_source.field1" (as expected), but other times it contains > a "\_source.field1", whose values are null. Why is the behavior not > consistent? Am I missing anything? Thank you! > > Cheers > Alex ---
**Author:** ![Alex\_2](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@Alex\_2](https://discuss.elastic.co/u/Alex_2)\ **Post date:** [September 27, 2011, 9:19pm UTC](https://discuss.elastic.co/t/null-values-for-optional-fields/5458/3 "2011-09-27T21:19:36Z")
Hi Shay > Not sure I understand, if optional\_field is missing, then you expect the > response _not_ to include field1? Of course, not ๐Ÿ™‚ I would expect null when I extract the optional\_field, but the behavior is a bit erratic, so I need to use the following line to extract the value val field = getResponse.field("\_source.optional\_field") if ((field != null) && (field.values() != null) && (field.values()(0) != null)) { Some(field.values()(0).asInstanceOf[String]) } else { None } the last line is the value returned (it's Scala, Some and None and related to [Option](http://www.scala-lang.org/api/current/scala/Option.html) ) Could you please point me to the docs related to the better field extraction support? Cheers Alex On Tuesday, September 27, 2011 11:03:50 PM UTC+2, kimchy wrote: > Btw, in master, there is much better support for extracting fields from > source, you no longer (and should not) use the \_source prefix when it comes > out (if you do, it will still use the scripting hack to fetch it). > > On Tue, Sep 27, 2011 at 5:59 AM, Alex [alessandro.us...@gmail.com](mailto:alessandro.us...@gmail.com) wrote: > > > Hi Shay > > > > I found a problem that arises when using this line: > > > > getResponse = client.prepareGet(INDEX\_NAME, TYPE, documentId) > > .setFields("\_source.field1", "\_source.field2", > > "\_source.optional\_field") > > .setOperationThreaded(false) > > .execute() > > .actionGet() > > > > If the optional\_field is missing, the response's fields sometime contains > > no "\_source.field1" (as expected), but other times it contains > > a "\_source.field1", whose values are null. Why is the behavior not > > consistent? Am I missing anything? Thank you! > > > > Cheers > > Alex ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 27, 2011, 9:23pm UTC](https://discuss.elastic.co/t/null-values-for-optional-fields/5458/4 "2011-09-27T21:23:07Z")
Now I am completely confused. This is what you said: > If the optional\_field is missing, the response's fields sometime contains > no "\_source.field1" (as expected), but other times it contains > a "\_source.field1", whose values are null. Why is the behavior not > consistent? Am I missing anything? Thank you! How does it relate to the optional field? And what you pasted below, which is basically checking if optional\_field is null or not, and has nothing to do with field1? On Wed, Sep 28, 2011 at 12:19 AM, Alex \<[alessandro.usseglioviretta@gmail.com](mailto:alessandro.usseglioviretta@gmail.com) > wrote: > Hi Shay > > > Not sure I understand, if optional\_field is missing, then you expect the > > response _not_ to include field1? > > Of course, not ๐Ÿ™‚ I would expect null when I extract the optional\_field, > but the behavior is a bit erratic, so I need to use the following line to > extract the value > > val field = getResponse.field("\_source.optional\_field") > if ((field != null) && (field.values() != null) && (field.values()(0) != > null)) { Some(field.values()(0).asInstanceOf[String]) } else { None } > > the last line is the value returned (it's Scala, Some and None and related > to [Option](http://www.scala-lang.org/api/current/scala/Option.html) ) > > Could you please point me to the docs related to the better field > extraction support? > > Cheers > Alex > > On Tuesday, September 27, 2011 11:03:50 PM UTC+2, kimchy wrote: > > > Btw, in master, there is much better support for extracting fields from > > source, you no longer (and should not) use the \_source prefix when it comes > > out (if you do, it will still use the scripting hack to fetch it). > > > > On Tue, Sep 27, 2011 at 5:59 AM, Alex [alessandro.us...@gmail.com](mailto:alessandro.us...@gmail.com) wrote: > > > > > Hi Shay > > > > > > I found a problem that arises when using this line: > > > > > > getResponse = client.prepareGet(INDEX\_NAME, TYPE, documentId) > > > .setFields("\_source.field1", "\_source.field2", > > > "\_source.optional\_field") > > > .setOperationThreaded(false) > > > .execute() > > > .actionGet() > > > > > > If the optional\_field is missing, the response's fields sometime contains > > > no "\_source.field1" (as expected), but other times it contains > > > a "\_source.field1", whose values are null. Why is the behavior not > > > consistent? Am I missing anything? Thank you! > > > > > > Cheers > > > Alex ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 27, 2011, 9:27pm UTC](https://discuss.elastic.co/t/null-values-for-optional-fields/5458/5 "2011-09-27T21:27:33Z")
And, regarding your test for null value, it will be fixed with the new support. No docs really needed for it, you use the same addFields with the actual field names without the \_source prefix. On the other hand, it probably make sense not to return script field (\_source.something) null values as part of the field values. I'll fix that. Still not getting the optional\_field relation to field1. On Wed, Sep 28, 2011 at 12:23 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > Now I am completely confused. This is what you said: > > > If the optional\_field is missing, the response's fields sometime contains > > no "\_source.field1" (as expected), but other times it contains > > a "\_source.field1", whose values are null. Why is the behavior not > > consistent? Am I missing anything? Thank you! > > How does it relate to the optional field? And what you pasted below, which > is basically checking if optional\_field is null or not, and has nothing to > do with field1? > > On Wed, Sep 28, 2011 at 12:19 AM, Alex \< > [alessandro.usseglioviretta@gmail.com](mailto:alessandro.usseglioviretta@gmail.com)\> wrote: > > > Hi Shay > > > > > Not sure I understand, if optional\_field is missing, then you expect the > > > response _not_ to include field1? > > > > Of course, not ๐Ÿ™‚ I would expect null when I extract the optional\_field, > > but the behavior is a bit erratic, so I need to use the following line to > > extract the value > > > > val field = getResponse.field("\_source.optional\_field") > > if ((field != null) && (field.values() != null) && (field.values()(0) != > > null)) { Some(field.values()(0).asInstanceOf[String]) } else { None } > > > > the last line is the value returned (it's Scala, Some and None and related > > to [Option](http://www.scala-lang.org/api/current/scala/Option.html) ) > > > > Could you please point me to the docs related to the better field > > extraction support? > > > > Cheers > > Alex > > > > On Tuesday, September 27, 2011 11:03:50 PM UTC+2, kimchy wrote: > > > > > Btw, in master, there is much better support for extracting fields from > > > source, you no longer (and should not) use the \_source prefix when it comes > > > out (if you do, it will still use the scripting hack to fetch it). > > > > > > On Tue, Sep 27, 2011 at 5:59 AM, Alex [alessandro.us...@gmail.com](mailto:alessandro.us...@gmail.com)wrote: > > > > > > > Hi Shay > > > > > > > > I found a problem that arises when using this line: > > > > > > > > getResponse = client.prepareGet(INDEX\_NAME, TYPE, documentId) > > > > .setFields("\_source.field1", "\_source.field2", > > > > "\_source.optional\_field") > > > > .setOperationThreaded(false) > > > > .execute() > > > > .actionGet() > > > > > > > > If the optional\_field is missing, the response's fields sometime > > > > contains no "\_source.field1" (as expected), but other times it contains > > > > a "\_source.field1", whose values are null. Why is the behavior not > > > > consistent? Am I missing anything? Thank you! > > > > > > > > Cheers > > > > Alex ---
**Author:** ![Alex\_2](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@Alex\_2](https://discuss.elastic.co/u/Alex_2)\ **Post date:** [September 27, 2011, 9:30pm UTC](https://discuss.elastic.co/t/null-values-for-optional-fields/5458/6 "2011-09-27T21:30:19Z")
sorry, my mistake... it was too late in the night. This is the correct version: If the optional\_field is missing, the response's fields sometime contains no "\_source.optional\_field" (as expected), but other times it contains a "\_source.optional\_field", whose values are null. Alex ---
**Author:** ![Alex\_2](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@Alex\_2](https://discuss.elastic.co/u/Alex_2)\ **Post date:** [September 27, 2011, 9:38pm UTC](https://discuss.elastic.co/t/null-values-for-optional-fields/5458/7 "2011-09-27T21:38:49Z")
related to your suggestion: Is there a GetRequestBuilder.addFields method? I cannot find it Alex ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 27, 2011, 10:20pm UTC](https://discuss.elastic.co/t/null-values-for-optional-fields/5458/8 "2011-09-27T22:20:19Z")
> If the optional\_field is missing, the response's fields sometime contains > no "\_source.optional\_field" (as expected), but other times it contains > a "\_source.optional\_field", whose values are null. Right, that sadly might happen with teh current way extracting fields from \_source works in 0.17, in upcoming 0.18, you just say: optional\_field, with no \_source prefix, and it will work as expected. > related to your suggestion: Is there a GetRequestBuilder.addFields method? > I cannot find it There is GetRequestBuilder#setFields..., probably better to also have addFields in a similar manner to the search request, will add it... On Wed, Sep 28, 2011 at 12:38 AM, Alex \<[alessandro.usseglioviretta@gmail.com](mailto:alessandro.usseglioviretta@gmail.com) > wrote: > related to your suggestion: Is there a GetRequestBuilder.addFields method? > I cannot find it > > Alex ---
**Author:** ![Alex\_2](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@Alex\_2](https://discuss.elastic.co/u/Alex_2)\ **Post date:** [September 27, 2011, 10:59pm UTC](https://discuss.elastic.co/t/null-values-for-optional-fields/5458/9 "2011-09-27T22:59:07Z")
alright, it's good to know that I'm not doing anything wrong ๐Ÿ™‚ I've put together a small piece of code to replicate a similar problem: getting a field without '\_source' prefix results in a NullPointerException (unless I'm missing something) Here is the code: [https://gist.github.com/1246499](https://gist.github.com/1246499) Cheers Alex ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 28, 2011, 12:43pm UTC](https://discuss.elastic.co/t/null-values-for-optional-fields/5458/10 "2011-09-28T12:43:34Z")
Can you gist the full stack trace of the NPE you get? On Wed, Sep 28, 2011 at 1:59 AM, Alex [alessandro.usseglioviretta@gmail.com](mailto:alessandro.usseglioviretta@gmail.com)wrote: > alright, it's good to know that I'm not doing anything wrong ๐Ÿ™‚ > > I've put together a small piece of code to replicate a similar problem: > getting a field without '\_source' prefix results in a NullPointerException > (unless I'm missing something) > > Here is the code: [.setFields("\_source.obj") vs .setFields("obj"): The latter results in NullPointerException ยท GitHub](https://gist.github.com/1246499) > > Cheers > Alex ---
**Author:** ![Alex\_2](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@Alex\_2](https://discuss.elastic.co/u/Alex_2)\ **Post date:** [September 29, 2011, 9:43pm UTC](https://discuss.elastic.co/t/null-values-for-optional-fields/5458/11 "2011-09-29T21:43:41Z")
Here it is > Alex ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 2, 2011, 8:38am UTC](https://discuss.elastic.co/t/null-values-for-optional-fields/5458/12 "2011-10-02T08:38:46Z")
The doesn't look like its coming from elasticsearch... On Fri, Sep 30, 2011 at 12:43 AM, Alex \<[alessandro.usseglioviretta@gmail.com](mailto:alessandro.usseglioviretta@gmail.com) > wrote: > Here it is > > [Stack trace for Test\_1 ยท GitHub](https://gist.github.com/1252017) > > Alex ---
**Author:** ![Alex\_2](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@Alex\_2](https://discuss.elastic.co/u/Alex_2)\ **Post date:** [October 2, 2011, 2:33pm UTC](https://discuss.elastic.co/t/null-values-for-optional-fields/5458/13 "2011-10-02T14:33:47Z")
indeed, the exception is not thrown within elasticsearch, but it is caused by the fact that .setFields("\_source.obj") works fine whereas .setFields( "obj") doesn't (it returns a null), although, If I understand you well, they should be equivalent and the latter should be preferred. Alex ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 2, 2011, 10:09pm UTC](https://discuss.elastic.co/t/null-values-for-optional-fields/5458/14 "2011-10-02T22:09:44Z")
I think I mentioned that the latter (using just obj) is an improvement that is currently in master (upcoming 0.18), and in 0.17, you have to use \_source.obj. Once 0.18 is out, you should use just "obj". On Sun, Oct 2, 2011 at 4:33 PM, Alex [alessandro.usseglioviretta@gmail.com](mailto:alessandro.usseglioviretta@gmail.com)wrote: > indeed, the exception is not thrown within elasticsearch, but it is caused > by the fact that .setFields("\_source.obj") works fine whereas .setFields( > "obj") doesn't (it returns a null), although, If I understand you well, > they should be equivalent and the latter should be preferred. > > Alex ---
**Author:** ![Alex\_2](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@Alex\_2](https://discuss.elastic.co/u/Alex_2)\ **Post date:** [October 2, 2011, 11:08pm UTC](https://discuss.elastic.co/t/null-values-for-optional-fields/5458/15 "2011-10-02T23:08:23Z")
ok, cool... I got a bit confused. Thank you for the help! Alex ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:53am UTC](https://discuss.elastic.co/t/null-values-for-optional-fields/5458/16 "2017-07-06T03:53:04Z")