Null values for optional fields

Alex_2 · September 27, 2011, 2:59am

Hi Shay

I found a problem that arises when using this line:

getResponse = client.prepareGet(INDEX_NAME, TYPE, documentId)
.setFields("_source.field1", "_source.field2",
"_source.optional_field")
.setOperationThreaded(false)
.execute()
.actionGet()

If the optional_field is missing, the response's fields sometime contains
no "_source.field1" (as expected), but other times it contains
a "_source.field1", whose values are null. Why is the behavior not
consistent? Am I missing anything? Thank you!

Cheers
Alex

kimchy · September 27, 2011, 9:03pm

Not sure I understand, if optional_field is missing, then you expect the
response not to include field1?

Btw, in master, there is much better support for extracting fields from
source, you no longer (and should not) use the _source prefix when it comes
out (if you do, it will still use the scripting hack to fetch it).

On Tue, Sep 27, 2011 at 5:59 AM, Alex
alessandro.usseglioviretta@gmail.comwrote:

Hi Shay

I found a problem that arises when using this line:

getResponse = client.prepareGet(INDEX_NAME, TYPE, documentId)
.setFields("_source.field1", "_source.field2",
"_source.optional_field")
.setOperationThreaded(false)
.execute()
.actionGet()

If the optional_field is missing, the response's fields sometime contains
no "_source.field1" (as expected), but other times it contains
a "_source.field1", whose values are null. Why is the behavior not
consistent? Am I missing anything? Thank you!

Cheers
Alex

Alex_2 · September 27, 2011, 9:19pm

Hi Shay

Not sure I understand, if optional_field is missing, then you expect the
response not to include field1?

Of course, not I would expect null when I extract the optional_field, but
the behavior is a bit erratic, so I need to use the following line to
extract the value

val field = getResponse.field("_source.optional_field")
if ((field != null) && (field.values() != null) && (field.values()(0) !=
null)) { Some(field.values()(0).asInstanceOf[String]) } else { None }

the last line is the value returned (it's Scala, Some and None and related
to Option )

Could you please point me to the docs related to the better field extraction
support?

Cheers
Alex

On Tuesday, September 27, 2011 11:03:50 PM UTC+2, kimchy wrote:

Btw, in master, there is much better support for extracting fields from
source, you no longer (and should not) use the _source prefix when it comes
out (if you do, it will still use the scripting hack to fetch it).

On Tue, Sep 27, 2011 at 5:59 AM, Alex alessandro.us...@gmail.com wrote:

Hi Shay

I found a problem that arises when using this line:

getResponse = client.prepareGet(INDEX_NAME, TYPE, documentId)
.setFields("_source.field1", "_source.field2",
"_source.optional_field")
.setOperationThreaded(false)
.execute()
.actionGet()

If the optional_field is missing, the response's fields sometime contains
no "_source.field1" (as expected), but other times it contains
a "_source.field1", whose values are null. Why is the behavior not
consistent? Am I missing anything? Thank you!

Cheers
Alex

kimchy · September 27, 2011, 9:23pm

Now I am completely confused. This is what you said:

If the optional_field is missing, the response's fields sometime contains
no "_source.field1" (as expected), but other times it contains
a "_source.field1", whose values are null. Why is the behavior not
consistent? Am I missing anything? Thank you!

How does it relate to the optional field? And what you pasted below, which
is basically checking if optional_field is null or not, and has nothing to
do with field1?

On Wed, Sep 28, 2011 at 12:19 AM, Alex <alessandro.usseglioviretta@gmail.com

wrote:

Hi Shay

Not sure I understand, if optional_field is missing, then you expect the
response not to include field1?

Of course, not I would expect null when I extract the optional_field,
but the behavior is a bit erratic, so I need to use the following line to
extract the value

val field = getResponse.field("_source.optional_field")
if ((field != null) && (field.values() != null) && (field.values()(0) !=
null)) { Some(field.values()(0).asInstanceOf[String]) } else { None }

the last line is the value returned (it's Scala, Some and None and related
to Option )

Could you please point me to the docs related to the better field
extraction support?

Cheers
Alex

On Tuesday, September 27, 2011 11:03:50 PM UTC+2, kimchy wrote:

Btw, in master, there is much better support for extracting fields from
source, you no longer (and should not) use the _source prefix when it comes
out (if you do, it will still use the scripting hack to fetch it).

On Tue, Sep 27, 2011 at 5:59 AM, Alex alessandro.us...@gmail.com wrote:

Hi Shay

I found a problem that arises when using this line:

getResponse = client.prepareGet(INDEX_NAME, TYPE, documentId)
.setFields("_source.field1", "_source.field2",
"_source.optional_field")
.setOperationThreaded(false)
.execute()
.actionGet()

If the optional_field is missing, the response's fields sometime contains
no "_source.field1" (as expected), but other times it contains
a "_source.field1", whose values are null. Why is the behavior not
consistent? Am I missing anything? Thank you!

Cheers
Alex

kimchy · September 27, 2011, 9:27pm

And, regarding your test for null value, it will be fixed with the new
support. No docs really needed for it, you use the same addFields with the
actual field names without the _source prefix.

On the other hand, it probably make sense not to return script field
(_source.something) null values as part of the field values. I'll fix that.

Still not getting the optional_field relation to field1.

On Wed, Sep 28, 2011 at 12:23 AM, Shay Banon kimchy@gmail.com wrote:

Now I am completely confused. This is what you said:

If the optional_field is missing, the response's fields sometime contains
no "_source.field1" (as expected), but other times it contains
a "_source.field1", whose values are null. Why is the behavior not
consistent? Am I missing anything? Thank you!

How does it relate to the optional field? And what you pasted below, which
is basically checking if optional_field is null or not, and has nothing to
do with field1?

On Wed, Sep 28, 2011 at 12:19 AM, Alex <
alessandro.usseglioviretta@gmail.com> wrote:

Hi Shay

Not sure I understand, if optional_field is missing, then you expect the
response not to include field1?

Of course, not I would expect null when I extract the optional_field,
but the behavior is a bit erratic, so I need to use the following line to
extract the value

val field = getResponse.field("_source.optional_field")
if ((field != null) && (field.values() != null) && (field.values()(0) !=
null)) { Some(field.values()(0).asInstanceOf[String]) } else { None }

the last line is the value returned (it's Scala, Some and None and related
to Option )

Could you please point me to the docs related to the better field
extraction support?

Cheers
Alex

On Tuesday, September 27, 2011 11:03:50 PM UTC+2, kimchy wrote:

Btw, in master, there is much better support for extracting fields from
source, you no longer (and should not) use the _source prefix when it comes
out (if you do, it will still use the scripting hack to fetch it).

On Tue, Sep 27, 2011 at 5:59 AM, Alex alessandro.us...@gmail.comwrote:

Hi Shay

I found a problem that arises when using this line:

getResponse = client.prepareGet(INDEX_NAME, TYPE, documentId)
.setFields("_source.field1", "_source.field2",
"_source.optional_field")
.setOperationThreaded(false)
.execute()
.actionGet()

If the optional_field is missing, the response's fields sometime
contains no "_source.field1" (as expected), but other times it contains
a "_source.field1", whose values are null. Why is the behavior not
consistent? Am I missing anything? Thank you!

Cheers
Alex

Alex_2 · September 27, 2011, 9:30pm

sorry, my mistake... it was too late in the night. This is the correct
version:

If the optional_field is missing, the response's fields sometime contains
no "_source.optional_field" (as expected), but other times it contains
a "_source.optional_field", whose values are null.

Alex

Alex_2 · September 27, 2011, 9:38pm

related to your suggestion: Is there a GetRequestBuilder.addFields method? I
cannot find it

Alex

kimchy · September 27, 2011, 10:20pm

If the optional_field is missing, the response's fields sometime contains
no "_source.optional_field" (as expected), but other times it contains
a "_source.optional_field", whose values are null.

Right, that sadly might happen with teh current way extracting fields from
_source works in 0.17, in upcoming 0.18, you just say: optional_field, with
no _source prefix, and it will work as expected.

related to your suggestion: Is there a GetRequestBuilder.addFields method?
I cannot find it

There is GetRequestBuilder#setFields..., probably better to also have
addFields in a similar manner to the search request, will add it...

On Wed, Sep 28, 2011 at 12:38 AM, Alex <alessandro.usseglioviretta@gmail.com

wrote:

related to your suggestion: Is there a GetRequestBuilder.addFields method?
I cannot find it

Alex

Alex_2 · September 27, 2011, 10:59pm

alright, it's good to know that I'm not doing anything wrong

I've put together a small piece of code to replicate a similar problem:
getting a field without '_source' prefix results in a NullPointerException
(unless I'm missing something)

Here is the code: https://gist.github.com/1246499

Cheers
Alex

kimchy · September 28, 2011, 12:43pm

Can you gist the full stack trace of the NPE you get?

On Wed, Sep 28, 2011 at 1:59 AM, Alex
alessandro.usseglioviretta@gmail.comwrote:

alright, it's good to know that I'm not doing anything wrong

I've put together a small piece of code to replicate a similar problem:
getting a field without '_source' prefix results in a NullPointerException
(unless I'm missing something)

Here is the code: .setFields("_source.obj") vs .setFields("obj"): The latter results in NullPointerException · GitHub

Cheers
Alex

Alex_2 · September 29, 2011, 9:43pm

Here it is

gist.github.com

https://gist.github.com/anonymous/1252017

Test_1-StackTrace

Response [{abc=123}]
[error] (run-main) java.lang.NullPointerException
java.lang.NullPointerException
	at test.Test_1$delayedInit$body.apply(Test_1.scala:41)
	at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
	at scala.App$$anonfun$main$1.apply(App.scala:60)
	at scala.App$$anonfun$main$1.apply(App.scala:60)
	at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
	at scala.collection.immutable.List.foreach(List.scala:45)

This file has been truncated. show original

Alex

kimchy · October 2, 2011, 8:38am

The doesn't look like its coming from elasticsearch...

On Fri, Sep 30, 2011 at 12:43 AM, Alex <alessandro.usseglioviretta@gmail.com

wrote:

Here it is

Stack trace for Test_1 · GitHub

Alex

Alex_2 · October 2, 2011, 2:33pm

indeed, the exception is not thrown within elasticsearch, but it is caused
by the fact that .setFields("_source.obj") works fine whereas .setFields(
"obj") doesn't (it returns a null), although, If I understand you well, they
should be equivalent and the latter should be preferred.

Alex

kimchy · October 2, 2011, 10:09pm

I think I mentioned that the latter (using just obj) is an improvement that
is currently in master (upcoming 0.18), and in 0.17, you have to use
_source.obj. Once 0.18 is out, you should use just "obj".

On Sun, Oct 2, 2011 at 4:33 PM, Alex
alessandro.usseglioviretta@gmail.comwrote:

indeed, the exception is not thrown within elasticsearch, but it is caused
by the fact that .setFields("_source.obj") works fine whereas .setFields(
"obj") doesn't (it returns a null), although, If I understand you well,
they should be equivalent and the latter should be preferred.

Alex

Alex_2 · October 2, 2011, 11:08pm

ok, cool... I got a bit confused. Thank you for the help!

Alex