Script terms facet in elasticsearch 0.90.0 returning incorrect results

https://gist.github.com/ejain/5507570 (0.20.6) vs
https://gist.github.com/ejain/5507571 (0.90.0)

Looks like time zone offsets in dates are now ignored, and dates are
assumed to be in the default time zone (i.e. Pacific Time for me) when
being accessed from a script?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

For the record: Dates in scripts now appear to use the default time zone;
previously they were always UTC. Not necessarily a bug, but a breaking
change to be aware of...

On Thursday, May 2, 2013 11:54:22 PM UTC-7, Eric Jain wrote:

Example request and (correct) response when using a script terms facet in elasticsearch 0.20.6 (+ lang-javascript 1.2.0). · GitHub (0.20.6) vs
Example request and (incorrect) response when using a script terms facet in elasticsearch 0.90.0 (+ lang-javascript 1.3.0). · GitHub (0.90.0)

Looks like time zone offsets in dates are now ignored, and dates are
assumed to be in the default time zone (i.e. Pacific Time for me) when
being accessed from a script?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks for this one. What happens if you give '2012-11-30T15:00:00Z'? The
given datetime is treated as UTC or default timezone too ?

2013/5/7 Eric Jain eric.jain@gmail.com

For the record: Dates in scripts now appear to use the default time zone;
previously they were always UTC. Not necessarily a bug, but a breaking
change to be aware of...

On Thursday, May 2, 2013 11:54:22 PM UTC-7, Eric Jain wrote:

https://gist.github.com/ejain/**5507570https://gist.github.com/ejain/5507570(0.20.6) vs
https://gist.github.com/ejain/**5507571https://gist.github.com/ejain/5507571(0.90.0)

Looks like time zone offsets in dates are now ignored, and dates are
assumed to be in the default time zone (i.e. Pacific Time for me) when
being accessed from a script?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Wed, May 8, 2013 at 5:37 AM, Marcin Dojwa m.dojwa@livechatinc.com wrote:

Thanks for this one. What happens if you give '2012-11-30T15:00:00Z'? The
given datetime is treated as UTC or default timezone too ?

If your local timezone offset is e.g. -07:00, doc['timestamp'].date is
now (0.90) '2012-11-30T08:00:00-07:00'. Previously (0.20), it would
have been '2012-11-30T15:00:00Z'; not because elasticsearch was
honoring the date's original timezone offset (I wish it did!), but
because the date property in a script would be always be UTC.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

For me it sounds like a bug. Did you report the issue here
Issues · elastic/elasticsearch · GitHub ?

2013/5/8 Eric Jain eric.jain@gmail.com

On Wed, May 8, 2013 at 5:37 AM, Marcin Dojwa m.dojwa@livechatinc.com
wrote:

Thanks for this one. What happens if you give '2012-11-30T15:00:00Z'? The
given datetime is treated as UTC or default timezone too ?

If your local timezone offset is e.g. -07:00, doc['timestamp'].date is
now (0.90) '2012-11-30T08:00:00-07:00'. Previously (0.20), it would
have been '2012-11-30T15:00:00Z'; not because elasticsearch was
honoring the date's original timezone offset (I wish it did!), but
because the date property in a script would be always be UTC.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Thu, May 9, 2013 at 1:45 AM, Marcin Dojwa m.dojwa@livechatinc.com wrote:

For me it sounds like a bug. Did you report the issue here
Issues · elastic/elasticsearch · GitHub ?

I haven't report the issue yet; I was hoping to hear from someone at
elasticsearch whether this change was intentional or not.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I suggest reporting an issue. If this is not an issue, you will get fast
response there. Could you give a link to the issue here when you report it
? Solving this problem is very important for me too before I deploy 0.90 on
production. Thanks.

2013/5/9 Eric Jain eric.jain@gmail.com

On Thu, May 9, 2013 at 1:45 AM, Marcin Dojwa m.dojwa@livechatinc.com
wrote:

For me it sounds like a bug. Did you report the issue here
Issues · elastic/elasticsearch · GitHub ?

I haven't report the issue yet; I was hoping to hear from someone at
elasticsearch whether this change was intentional or not.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On 5/8/2013 9:29 AM, Eric Jain wrote:

On Wed, May 8, 2013 at 5:37 AM, Marcin Dojwa m.dojwa@livechatinc.com wrote:

Thanks for this one. What happens if you give '2012-11-30T15:00:00Z'? The
given datetime is treated as UTC or default timezone too ?
If your local timezone offset is e.g. -07:00, doc['timestamp'].date is
now (0.90) '2012-11-30T08:00:00-07:00'. Previously (0.20), it would
have been '2012-11-30T15:00:00Z'; not because elasticsearch was
honoring the date's original timezone offset (I wish it did!), but
because the date property in a script would be always be UTC.

Actually both of those are the same millisecond value stored in the date
field displayed differently (In the Summer 8 AM in Seattle is the same
moment in time as 3 PM in London) Therefore, it depends on the
definition of what the script aDateField.date is supposed to correspond
to in the Joda date library and thus what toString method is used to
convert the results back to the caller. It is not a matter of honoring
the original TZ, because it can't, the field stores a millisecond value;
that's it. On the way out nothing knows how it got that millisecond
value into the field.

As Eric as observed
it WAS converting the millisecond value to the result string using UTC.
but now
it IS converting to the millisecond value to the result string using the
DEFAULT VM timezone.

"The default date parsing used if no format is specified is
ISODateTimeFormat.dateOptionalTimeParser
http://joda-time.sourceforge.net/api-release/org/joda/time/format/ISODateTimeFormat.html#dateOptionalTimeParser()."
"... The first format will also act as the one that converts back from
milliseconds to a string representation. ..."

-- Elasticsearch Platform — Find real-time answers at scale | Elastic

None of the formats specified make any claim as to what TZ will be used
when formatting the millisecond value into a readable string, but
certainly being able to pick one by setting default VM timezone is a
valid choice, but so is the earlier idea of always using UTC, so I don't
know which behavior is intended. It is also important that all output
formats do the same thing and not mix VM default with UTC. One idea is
that forcing UTC as the default on the way out reminds folks to think
about what they want to see and to learn the difference between how it
is stored and how it is transported back (hopefully it is short leap of
logic to thinking about how it is transported into the index).

Just as a warning to all, having client and server us different
timezones can make hard to find bugs when using the default input format
which allows dates (without times) sent from clients generating local
dates and then falling into the previous or next day, because they end
up at "somebody else's" (the servers) midnight when it is converted for
storage into a millisecond value.
Client says "2013-05-10" (thinking that means "2012-05-10 T 00:00:00
-07:00" or "2012-05-10 T 07:00:00Z"), server records a timestamp of 7
hours earlier "2012-05-10 T 00:00:00 Z". If later the client code
fully specifies things with times and timezones, things can be curiously
mismatched with the original timestamp and get subtly confusing.

-Paul

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Thursday, May 9, 2013 1:45:13 AM UTC-7, Marcin Dojwa wrote:

For me it sounds like a bug. Did you report the issue here
Issues · elastic/elasticsearch · GitHub ?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.