Null values for optional fields


(Alex-2) #1

Hi Shay

I found a problem that arises when using this line:

getResponse = client.prepareGet(INDEX_NAME, TYPE, documentId)
.setFields("_source.field1", "_source.field2",
"_source.optional_field")
.setOperationThreaded(false)
.execute()
.actionGet()

If the optional_field is missing, the response's fields sometime contains
no "_source.field1" (as expected), but other times it contains
a "_source.field1", whose values are null. Why is the behavior not
consistent? Am I missing anything? Thank you!

Cheers
Alex


(Shay Banon) #2

Not sure I understand, if optional_field is missing, then you expect the
response not to include field1?

Btw, in master, there is much better support for extracting fields from
source, you no longer (and should not) use the _source prefix when it comes
out (if you do, it will still use the scripting hack to fetch it).

On Tue, Sep 27, 2011 at 5:59 AM, Alex
alessandro.usseglioviretta@gmail.comwrote:

Hi Shay

I found a problem that arises when using this line:

getResponse = client.prepareGet(INDEX_NAME, TYPE, documentId)
.setFields("_source.field1", "_source.field2",
"_source.optional_field")
.setOperationThreaded(false)
.execute()
.actionGet()

If the optional_field is missing, the response's fields sometime contains
no "_source.field1" (as expected), but other times it contains
a "_source.field1", whose values are null. Why is the behavior not
consistent? Am I missing anything? Thank you!

Cheers
Alex


(Alex-2) #3

Hi Shay

Not sure I understand, if optional_field is missing, then you expect the
response not to include field1?

Of course, not :slight_smile: I would expect null when I extract the optional_field, but
the behavior is a bit erratic, so I need to use the following line to
extract the value

val field = getResponse.field("_source.optional_field")
if ((field != null) && (field.values() != null) && (field.values()(0) !=
null)) { Some(field.values()(0).asInstanceOf[String]) } else { None }

the last line is the value returned (it's Scala, Some and None and related
to http://www.scala-lang.org/api/current/scala/Option.html )

Could you please point me to the docs related to the better field extraction
support?

Cheers
Alex

On Tuesday, September 27, 2011 11:03:50 PM UTC+2, kimchy wrote:

Btw, in master, there is much better support for extracting fields from
source, you no longer (and should not) use the _source prefix when it comes
out (if you do, it will still use the scripting hack to fetch it).

On Tue, Sep 27, 2011 at 5:59 AM, Alex alessandro.us...@gmail.com wrote:

Hi Shay

I found a problem that arises when using this line:

getResponse = client.prepareGet(INDEX_NAME, TYPE, documentId)
.setFields("_source.field1", "_source.field2",
"_source.optional_field")
.setOperationThreaded(false)
.execute()
.actionGet()

If the optional_field is missing, the response's fields sometime contains
no "_source.field1" (as expected), but other times it contains
a "_source.field1", whose values are null. Why is the behavior not
consistent? Am I missing anything? Thank you!

Cheers
Alex


(Shay Banon) #4

Now I am completely confused. This is what you said:

If the optional_field is missing, the response's fields sometime contains
no "_source.field1" (as expected), but other times it contains
a "_source.field1", whose values are null. Why is the behavior not
consistent? Am I missing anything? Thank you!

How does it relate to the optional field? And what you pasted below, which
is basically checking if optional_field is null or not, and has nothing to
do with field1?

On Wed, Sep 28, 2011 at 12:19 AM, Alex <alessandro.usseglioviretta@gmail.com

wrote:

Hi Shay

Not sure I understand, if optional_field is missing, then you expect the
response not to include field1?

Of course, not :slight_smile: I would expect null when I extract the optional_field,
but the behavior is a bit erratic, so I need to use the following line to
extract the value

val field = getResponse.field("_source.optional_field")
if ((field != null) && (field.values() != null) && (field.values()(0) !=
null)) { Some(field.values()(0).asInstanceOf[String]) } else { None }

the last line is the value returned (it's Scala, Some and None and related
to http://www.scala-lang.org/api/current/scala/Option.html )

Could you please point me to the docs related to the better field
extraction support?

Cheers
Alex

On Tuesday, September 27, 2011 11:03:50 PM UTC+2, kimchy wrote:

Btw, in master, there is much better support for extracting fields from
source, you no longer (and should not) use the _source prefix when it comes
out (if you do, it will still use the scripting hack to fetch it).

On Tue, Sep 27, 2011 at 5:59 AM, Alex alessandro.us...@gmail.com wrote:

Hi Shay

I found a problem that arises when using this line:

getResponse = client.prepareGet(INDEX_NAME, TYPE, documentId)
.setFields("_source.field1", "_source.field2",
"_source.optional_field")
.setOperationThreaded(false)
.execute()
.actionGet()

If the optional_field is missing, the response's fields sometime contains
no "_source.field1" (as expected), but other times it contains
a "_source.field1", whose values are null. Why is the behavior not
consistent? Am I missing anything? Thank you!

Cheers
Alex


(Shay Banon) #5

And, regarding your test for null value, it will be fixed with the new
support. No docs really needed for it, you use the same addFields with the
actual field names without the _source prefix.

On the other hand, it probably make sense not to return script field
(_source.something) null values as part of the field values. I'll fix that.

Still not getting the optional_field relation to field1.

On Wed, Sep 28, 2011 at 12:23 AM, Shay Banon kimchy@gmail.com wrote:

Now I am completely confused. This is what you said:

If the optional_field is missing, the response's fields sometime contains
no "_source.field1" (as expected), but other times it contains
a "_source.field1", whose values are null. Why is the behavior not
consistent? Am I missing anything? Thank you!

How does it relate to the optional field? And what you pasted below, which
is basically checking if optional_field is null or not, and has nothing to
do with field1?

On Wed, Sep 28, 2011 at 12:19 AM, Alex <
alessandro.usseglioviretta@gmail.com> wrote:

Hi Shay

Not sure I understand, if optional_field is missing, then you expect the
response not to include field1?

Of course, not :slight_smile: I would expect null when I extract the optional_field,
but the behavior is a bit erratic, so I need to use the following line to
extract the value

val field = getResponse.field("_source.optional_field")
if ((field != null) && (field.values() != null) && (field.values()(0) !=
null)) { Some(field.values()(0).asInstanceOf[String]) } else { None }

the last line is the value returned (it's Scala, Some and None and related
to http://www.scala-lang.org/api/current/scala/Option.html )

Could you please point me to the docs related to the better field
extraction support?

Cheers
Alex

On Tuesday, September 27, 2011 11:03:50 PM UTC+2, kimchy wrote:

Btw, in master, there is much better support for extracting fields from
source, you no longer (and should not) use the _source prefix when it comes
out (if you do, it will still use the scripting hack to fetch it).

On Tue, Sep 27, 2011 at 5:59 AM, Alex alessandro.us...@gmail.comwrote:

Hi Shay

I found a problem that arises when using this line:

getResponse = client.prepareGet(INDEX_NAME, TYPE, documentId)
.setFields("_source.field1", "_source.field2",
"_source.optional_field")
.setOperationThreaded(false)
.execute()
.actionGet()

If the optional_field is missing, the response's fields sometime
contains no "_source.field1" (as expected), but other times it contains
a "_source.field1", whose values are null. Why is the behavior not
consistent? Am I missing anything? Thank you!

Cheers
Alex


(Alex-2) #6

sorry, my mistake... it was too late in the night. This is the correct
version:

If the optional_field is missing, the response's fields sometime contains
no "_source.optional_field" (as expected), but other times it contains
a "_source.optional_field", whose values are null.

Alex


(Alex-2) #7

related to your suggestion: Is there a GetRequestBuilder.addFields method? I
cannot find it

Alex


(Shay Banon) #8

If the optional_field is missing, the response's fields sometime contains
no "_source.optional_field" (as expected), but other times it contains
a "_source.optional_field", whose values are null.

Right, that sadly might happen with teh current way extracting fields from
_source works in 0.17, in upcoming 0.18, you just say: optional_field, with
no _source prefix, and it will work as expected.

related to your suggestion: Is there a GetRequestBuilder.addFields method?
I cannot find it

There is GetRequestBuilder#setFields..., probably better to also have
addFields in a similar manner to the search request, will add it...

On Wed, Sep 28, 2011 at 12:38 AM, Alex <alessandro.usseglioviretta@gmail.com

wrote:

related to your suggestion: Is there a GetRequestBuilder.addFields method?
I cannot find it

Alex


(Alex-2) #9

alright, it's good to know that I'm not doing anything wrong :slight_smile:

I've put together a small piece of code to replicate a similar problem:
getting a field without '_source' prefix results in a NullPointerException
(unless I'm missing something)

Here is the code: https://gist.github.com/1246499

Cheers
Alex


(Shay Banon) #10

Can you gist the full stack trace of the NPE you get?

On Wed, Sep 28, 2011 at 1:59 AM, Alex
alessandro.usseglioviretta@gmail.comwrote:

alright, it's good to know that I'm not doing anything wrong :slight_smile:

I've put together a small piece of code to replicate a similar problem:
getting a field without '_source' prefix results in a NullPointerException
(unless I'm missing something)

Here is the code: https://gist.github.com/1246499

Cheers
Alex


(Alex-2) #11

Here it is

Alex


(Shay Banon) #12

The doesn't look like its coming from elasticsearch...

On Fri, Sep 30, 2011 at 12:43 AM, Alex <alessandro.usseglioviretta@gmail.com

wrote:

Here it is

https://gist.github.com/1252017

Alex


(Alex-2) #13

indeed, the exception is not thrown within elasticsearch, but it is caused
by the fact that .setFields("_source.obj") works fine whereas .setFields(
"obj") doesn't (it returns a null), although, If I understand you well, they
should be equivalent and the latter should be preferred.

Alex


(Shay Banon) #14

I think I mentioned that the latter (using just obj) is an improvement that
is currently in master (upcoming 0.18), and in 0.17, you have to use
_source.obj. Once 0.18 is out, you should use just "obj".

On Sun, Oct 2, 2011 at 4:33 PM, Alex
alessandro.usseglioviretta@gmail.comwrote:

indeed, the exception is not thrown within elasticsearch, but it is caused
by the fact that .setFields("_source.obj") works fine whereas .setFields(
"obj") doesn't (it returns a null), although, If I understand you well,
they should be equivalent and the latter should be preferred.

Alex


(Alex-2) #15

ok, cool... I got a bit confused. Thank you for the help!

Alex


(system) #16