Scripting and dates

Michael_Giagnocavo · September 10, 2014, 6:02am

I'm trying to work with dates inside a script. I've got a few questions:

How do I reference a script that I have in the scripts directory? Simply POSTing to /index/type/id/_update with { "script": "scriptname" } does not seem to work. "No such property: scriptname for class: ScriptN", where N starts at 3 (I have two .groovy files in my scripts directory).

2: How can I get actual date objects from the source? ctx._source.fieldname always returns a type string, even if I just created the field with ctx._source.fieldname = new Date(). Right now I'm parsing the string output in Groovy, which seems suboptimal.

3: Are ISO8601 dates not fully supported, as far as arbitrary fractional second decimals? (Not just 3 or another fixed number?) Any suggestions on handling JSON input from multiple sources, some of which have high-precision?

4: Can I use a script to project the document into a scalar for aggregates? For instance, if I have Date fields "start" and "end", and want to calculate the average duration (start - end) in an aggregate. I see value-level scripts are allowed, and 1.4 has "scripted metric aggregation". For now am I best off just storing the duration in the document?

Thank you,
Michael

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/13d4cf783a83447a84b62206605ad312%40CO1PR07MB331.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

vineeth_mohan_2 · September 10, 2014, 11:13am

Hello Michael ,

Please find the answers in the order of questions you have asked -

Referencing script from file system is explained here. It has very
well worked for me , please double check on it -
Elasticsearch Platform — Find real-time answers at scale | Elastic
I feel you haven't declared that field as date type in the schema .
If you had done that , you will recieve the epoch instead of string. -
Elasticsearch Platform — Find real-time answers at scale | Elastic
Dates are internally stored as epoch. So it should handle that second
fraction too. More on the format can be seen here -
DateTimeFormat (Joda time 2.1 API)
What exactly do you want to do with the duration ? If its range
aggregation , it does have script support -
Elasticsearch Platform — Find real-time answers at scale | Elastic

Thanks
Vineeth

On Wed, Sep 10, 2014 at 11:32 AM, Michael Giagnocavo mgg@giagnocavo.net
wrote:

I'm trying to work with dates inside a script. I've got a few questions:

How do I reference a script that I have in the scripts directory?
Simply POSTing to /index/type/id/_update with { "script": "scriptname" }
does not seem to work. "No such property: scriptname for class: ScriptN",
where N starts at 3 (I have two .groovy files in my scripts directory).

2: How can I get actual date objects from the source?
ctx._source.fieldname always returns a type string, even if I just created
the field with ctx._source.fieldname = new Date(). Right now I'm parsing
the string output in Groovy, which seems suboptimal.

3: Are ISO8601 dates not fully supported, as far as arbitrary fractional
second decimals? (Not just 3 or another fixed number?) Any suggestions on
handling JSON input from multiple sources, some of which have
high-precision?

4: Can I use a script to project the document into a scalar for
aggregates? For instance, if I have Date fields "start" and "end", and want
to calculate the average duration (start - end) in an aggregate. I see
value-level scripts are allowed, and 1.4 has "scripted metric aggregation".
For now am I best off just storing the duration in the document?

Thank you,
Michael

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/13d4cf783a83447a84b62206605ad312%40CO1PR07MB331.namprd07.prod.outlook.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Michael_Giagnocavo · September 10, 2014, 5:56pm

Thanks for the reply Vineeth. Here’s a concise example showing the date type problem. Despite having a mapping, the script is getting the properties as strings. How can I verify how ElasticSearch is actually storing something?

$ cat /etc/elasticsearch/scripts/setdur.groovy
ctx._source.dur = (ctx._source.end.getTime() - ctx._source.start.getTime())

$ curl -i -XPUT http://localhost:9200/tesi/ -d '{
"mappings": {
"testt": { "properties": { "start": { "type": "date" }, "end": { "type": "date" } } }
}
}'
HTTP/1.1 200 OK

$ curl -i -XPOST http://localhost:9200/tesi/testt/1 -d '{ "start": "2014-09-01T12:00:00", "end": "2014-09-02T12:00:00" }'
HTTP/1.1 201 Created

rooty@sofab-es1:~$ curl -i -XPOST http://localhost:9200/tesi/testt/1/_update -d '{"script": "setdur", "lang": "groovy"}'
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=UTF-8
Content-Length: 399

{"error":"ElasticsearchIllegalArgumentException[failed to execute script]; nested: GroovyScriptExecutionException[MissingMethodException[No signature of method: java.lang.String.getTime() is applicable for argument types: () values: []\nPossible solutions: getBytes(), trim(), getBytes(java.lang.String), getBytes(java.nio.charset.Charset), getAt(groovy.lang.IntRange), getAt(int)]]; ","status":400}rooty@sofab-es1:~$

Any suggestions?
-Michael

From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of vineeth mohan
Sent: Wednesday, September 10, 2014 5:14 AM
To: elasticsearch@googlegroups.com
Subject: Re: Scripting and dates

Hello Michael ,

Please find the answers in the order of questions you have asked -

Referencing script from file system is explained here. It has very well worked for me , please double check on it - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html
I feel you haven't declared that field as date type in the schema . If you had done that , you will recieve the epoch instead of string. - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date
Dates are internally stored as epoch. So it should handle that second fraction too. More on the format can be seen here - http://joda-time.sourceforge.net/api-release/org/joda/time/format/DateTimeFormat.html
What exactly do you want to do with the duration ? If its range aggregation , it does have script support - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-range-aggregation.html#search-aggregations-bucket-range-aggregation
Thanks
Vineeth

On Wed, Sep 10, 2014 at 11:32 AM, Michael Giagnocavo <mgg@giagnocavo.net mailto:mgg@giagnocavo.net> wrote:
I'm trying to work with dates inside a script. I've got a few questions:

How do I reference a script that I have in the scripts directory? Simply POSTing to /index/type/id/_update with { "script": "scriptname" } does not seem to work. "No such property: scriptname for class: ScriptN", where N starts at 3 (I have two .groovy files in my scripts directory).

2: How can I get actual date objects from the source? ctx._source.fieldname always returns a type string, even if I just created the field with ctx._source.fieldname = new Date(). Right now I'm parsing the string output in Groovy, which seems suboptimal.

3: Are ISO8601 dates not fully supported, as far as arbitrary fractional second decimals? (Not just 3 or another fixed number?) Any suggestions on handling JSON input from multiple sources, some of which have high-precision?

4: Can I use a script to project the document into a scalar for aggregates? For instance, if I have Date fields "start" and "end", and want to calculate the average duration (start - end) in an aggregate. I see value-level scripts are allowed, and 1.4 has "scripted metric aggregation". For now am I best off just storing the duration in the document?

Thank you,
Michael

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch%2Bunsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/13d4cf783a83447a84b62206605ad312%40CO1PR07MB331.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/06f511148cb64db49990663de158e9b6%40CO1PR07MB331.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

Michael_Giagnocavo · September 10, 2014, 6:42pm

BTW I found the problem with referring to a script by name. If the script has an error, then it fails on compile, written to error log. It’s then not considered a script. ES might want to change that behavior, so if you use “script” : “brokenscript” you get an error indicating what’s actually wrong. Of course if you know about this behavior I guess it’s not a big deal.

From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of vineeth mohan
Sent: Wednesday, September 10, 2014 5:14 AM
To: elasticsearch@googlegroups.com
Subject: Re: Scripting and dates

Hello Michael ,

Please find the answers in the order of questions you have asked -

Referencing script from file system is explained here. It has very well worked for me , please double check on it - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html
I feel you haven't declared that field as date type in the schema . If you had done that , you will recieve the epoch instead of string. - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date
Dates are internally stored as epoch. So it should handle that second fraction too. More on the format can be seen here - http://joda-time.sourceforge.net/api-release/org/joda/time/format/DateTimeFormat.html
What exactly do you want to do with the duration ? If its range aggregation , it does have script support - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-range-aggregation.html#search-aggregations-bucket-range-aggregation
Thanks
Vineeth

On Wed, Sep 10, 2014 at 11:32 AM, Michael Giagnocavo <mgg@giagnocavo.net mailto:mgg@giagnocavo.net> wrote:
I'm trying to work with dates inside a script. I've got a few questions:

How do I reference a script that I have in the scripts directory? Simply POSTing to /index/type/id/_update with { "script": "scriptname" } does not seem to work. "No such property: scriptname for class: ScriptN", where N starts at 3 (I have two .groovy files in my scripts directory).

2: How can I get actual date objects from the source? ctx._source.fieldname always returns a type string, even if I just created the field with ctx._source.fieldname = new Date(). Right now I'm parsing the string output in Groovy, which seems suboptimal.

3: Are ISO8601 dates not fully supported, as far as arbitrary fractional second decimals? (Not just 3 or another fixed number?) Any suggestions on handling JSON input from multiple sources, some of which have high-precision?

4: Can I use a script to project the document into a scalar for aggregates? For instance, if I have Date fields "start" and "end", and want to calculate the average duration (start - end) in an aggregate. I see value-level scripts are allowed, and 1.4 has "scripted metric aggregation". For now am I best off just storing the duration in the document?

Thank you,
Michael

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch%2Bunsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/13d4cf783a83447a84b62206605ad312%40CO1PR07MB331.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ef8054ee424145a489ee496117ae0dd4%40CO1PR07MB331.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

vineeth_mohan_2 · September 10, 2014, 8:13pm

Hello Michael ,

This should work for you -

cat scr
{
"script": "sdf = new
java.text.SimpleDateFormat('yyyy-MM-dd\'T\'HH:mm:ss');startDate =
sdf.parse(ctx._source.start);endDate = sdf.parse(ctx._source.end);
ctx._source.diff = endDate.getTime() - startDate.getTime();"
}
curl -XPOST http://localhost:9200/test/logs/1/_update -d @scr
{"_index":"test","_type":"logs","_id":"1","_version":10

I was in the pretext that one can access the doc object for updation , but
seems we can only access _source here.
In that case , you need to parse the string to date object and then do the
stuffs.
The above works for me perfectly.

Thanks
Vineeth

On Thu, Sep 11, 2014 at 12:12 AM, Michael Giagnocavo mgg@giagnocavo.net
wrote:

BTW I found the problem with referring to a script by name. If the
script has an error, then it fails on compile, written to error log. It’s
then not considered a script. ES might want to change that behavior, so if
you use “script” : “brokenscript” you get an error indicating what’s
actually wrong. Of course if you know about this behavior I guess it’s not
a big deal.

From: elasticsearch@googlegroups.com [mailto:
elasticsearch@googlegroups.com] *On Behalf Of *vineeth mohan
Sent: Wednesday, September 10, 2014 5:14 AM
To: elasticsearch@googlegroups.com
Subject: Re: Scripting and dates

Hello Michael ,

Please find the answers in the order of questions you have asked -

Referencing script from file system is explained here. It has very
well worked for me , please double check on it -
Elasticsearch Platform — Find real-time answers at scale | Elastic

I feel you haven't declared that field as date type in the schema .
If you had done that , you will recieve the epoch instead of string. -
Elasticsearch Platform — Find real-time answers at scale | Elastic

Dates are internally stored as epoch. So it should handle that
second fraction too. More on the format can be seen here -
DateTimeFormat (Joda time 2.1 API)

What exactly do you want to do with the duration ? If its range
aggregation , it does have script support -
Elasticsearch Platform — Find real-time answers at scale | Elastic

Thanks
      Vineeth
On Wed, Sep 10, 2014 at 11:32 AM, Michael Giagnocavo mgg@giagnocavo.net
wrote:

I'm trying to work with dates inside a script. I've got a few questions:

How do I reference a script that I have in the scripts directory?
Simply POSTing to /index/type/id/_update with { "script": "scriptname" }
does not seem to work. "No such property: scriptname for class: ScriptN",
where N starts at 3 (I have two .groovy files in my scripts directory).

2: How can I get actual date objects from the source?
ctx._source.fieldname always returns a type string, even if I just created
the field with ctx._source.fieldname = new Date(). Right now I'm parsing
the string output in Groovy, which seems suboptimal.

3: Are ISO8601 dates not fully supported, as far as arbitrary fractional
second decimals? (Not just 3 or another fixed number?) Any suggestions on
handling JSON input from multiple sources, some of which have
high-precision?

4: Can I use a script to project the document into a scalar for
aggregates? For instance, if I have Date fields "start" and "end", and want
to calculate the average duration (start - end) in an aggregate. I see
value-level scripts are allowed, and 1.4 has "scripted metric aggregation".
For now am I best off just storing the duration in the document?

Thank you,
Michael

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/13d4cf783a83447a84b62206605ad312%40CO1PR07MB331.namprd07.prod.outlook.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ef8054ee424145a489ee496117ae0dd4%40CO1PR07MB331.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/ef8054ee424145a489ee496117ae0dd4%40CO1PR07MB331.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5nk9KVGSHZHHx8OUyvGaq6MknOim23%2BBaNnZfw3TEuRDg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Michael_Giagnocavo · September 10, 2014, 11:46pm

Thank you, I did have to resort to parsing the date string. The next problem is that dates don’t seem to come back in a fixed format. If I post a date with just the date string, that’s what I get back. Likewise for fractional seconds. So I do some string work with substring first.

How can I determine if ES has actually indexed a value as a date?

-Michael

From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of vineeth mohan
Sent: Wednesday, September 10, 2014 2:13 PM
To: elasticsearch@googlegroups.com
Subject: Re: Scripting and dates

Hello Michael ,

This should work for you -

cat scr
{
"script": "sdf = new java.text.SimpleDateFormat('yyyy-MM-dd\'T\'HH:mm:ss');startDate = sdf.parse(ctx._source.start);endDate = sdf.parse(ctx._source.end); ctx._source.diff = endDate.getTime() - startDate.getTime();"
}
curl -XPOST http://localhost:9200/test/logs/1/_update -d @scr
{"_index":"test","_type":"logs","_id":"1","_version":10

I was in the pretext that one can access the doc object for updation , but seems we can only access _source here.
In that case , you need to parse the string to date object and then do the stuffs.
The above works for me perfectly.

Thanks
Vineeth

On Thu, Sep 11, 2014 at 12:12 AM, Michael Giagnocavo <mgg@giagnocavo.net mailto:mgg@giagnocavo.net> wrote:
BTW I found the problem with referring to a script by name. If the script has an error, then it fails on compile, written to error log. It’s then not considered a script. ES might want to change that behavior, so if you use “script” : “brokenscript” you get an error indicating what’s actually wrong. Of course if you know about this behavior I guess it’s not a big deal.

From: elasticsearch@googlegroups.com mailto:elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com mailto:elasticsearch@googlegroups.com] On Behalf Of vineeth mohan
Sent: Wednesday, September 10, 2014 5:14 AM
To: elasticsearch@googlegroups.com mailto:elasticsearch@googlegroups.com
Subject: Re: Scripting and dates

Hello Michael ,

Please find the answers in the order of questions you have asked -

Referencing script from file system is explained here. It has very well worked for me , please double check on it - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html
I feel you haven't declared that field as date type in the schema . If you had done that , you will recieve the epoch instead of string. - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date
Dates are internally stored as epoch. So it should handle that second fraction too. More on the format can be seen here - http://joda-time.sourceforge.net/api-release/org/joda/time/format/DateTimeFormat.html
What exactly do you want to do with the duration ? If its range aggregation , it does have script support - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-range-aggregation.html#search-aggregations-bucket-range-aggregation
Thanks
Vineeth

On Wed, Sep 10, 2014 at 11:32 AM, Michael Giagnocavo <mgg@giagnocavo.net mailto:mgg@giagnocavo.net> wrote:
I'm trying to work with dates inside a script. I've got a few questions:

How do I reference a script that I have in the scripts directory? Simply POSTing to /index/type/id/_update with { "script": "scriptname" } does not seem to work. "No such property: scriptname for class: ScriptN", where N starts at 3 (I have two .groovy files in my scripts directory).

2: How can I get actual date objects from the source? ctx._source.fieldname always returns a type string, even if I just created the field with ctx._source.fieldname = new Date(). Right now I'm parsing the string output in Groovy, which seems suboptimal.

3: Are ISO8601 dates not fully supported, as far as arbitrary fractional second decimals? (Not just 3 or another fixed number?) Any suggestions on handling JSON input from multiple sources, some of which have high-precision?

4: Can I use a script to project the document into a scalar for aggregates? For instance, if I have Date fields "start" and "end", and want to calculate the average duration (start - end) in an aggregate. I see value-level scripts are allowed, and 1.4 has "scripted metric aggregation". For now am I best off just storing the duration in the document?

Thank you,
Michael

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch%2Bunsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/13d4cf783a83447a84b62206605ad312%40CO1PR07MB331.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ef8054ee424145a489ee496117ae0dd4%40CO1PR07MB331.namprd07.prod.outlook.com https://groups.google.com/d/msgid/elasticsearch/ef8054ee424145a489ee496117ae0dd4%40CO1PR07MB331.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5nk9KVGSHZHHx8OUyvGaq6MknOim23%2BBaNnZfw3TEuRDg%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAGdPd5nk9KVGSHZHHx8OUyvGaq6MknOim23%2BBaNnZfw3TEuRDg%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2686ebcc594b43049c67fa21419f5555%40CO1PR07MB331.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

vineeth_mohan_2 · September 11, 2014, 3:38am

Hello Michael ,

When you set the type of the field as time , its indexed as time ( as
epoch to be more precise ).
And by default , it takes multiple date formats.
But the issue here is that , the index data is not available to you while
updating.
Only the stored data which is the date string is available.

So there are 2 options here

While declaring a field as type date , you can also specify the
format in which the date string comes in. Any other format is rejected.
This way hard coding the format in the script should work fine.
Store the epoch representation as time instead of date string. This
wont in anyway hinter with any of date operation like bucketing ,range
selection but at the same time , your update operations would be smooth.

Thanks
Vineeth

On Thu, Sep 11, 2014 at 5:16 AM, Michael Giagnocavo mgg@giagnocavo.net
wrote:

Thank you, I did have to resort to parsing the date string. The next
problem is that dates don’t seem to come back in a fixed format. If I post
a date with just the date string, that’s what I get back. Likewise for
fractional seconds. So I do some string work with substring first.

How can I determine if ES has actually indexed a value as a date?

-Michael

From: elasticsearch@googlegroups.com [mailto:
elasticsearch@googlegroups.com] *On Behalf Of *vineeth mohan
Sent: Wednesday, September 10, 2014 2:13 PM

To: elasticsearch@googlegroups.com
Subject: Re: Scripting and dates

Hello Michael ,

This should work for you -

cat scr

{
        "script": "sdf = new
java.text.SimpleDateFormat('yyyy-MM-dd\'T\'HH:mm:ss');startDate =
sdf.parse(ctx._source.start);endDate = sdf.parse(ctx._source.end);
ctx._source.diff = endDate.getTime() - startDate.getTime();"

}

curl -XPOST http://localhost:9200/test/logs/1/_update -d @scr

{"_index":"test","_type":"logs","_id":"1","_version":10

I was in the pretext that one can access the doc object for updation , but
seems we can only access _source here.

In that case , you need to parse the string to date object and then do the
stuffs.

The above works for me perfectly.

Thanks
       Vineeth
On Thu, Sep 11, 2014 at 12:12 AM, Michael Giagnocavo mgg@giagnocavo.net
wrote:

BTW I found the problem with referring to a script by name. If the
script has an error, then it fails on compile, written to error log. It’s
then not considered a script. ES might want to change that behavior, so if
you use “script” : “brokenscript” you get an error indicating what’s
actually wrong. Of course if you know about this behavior I guess it’s not
a big deal.

From: elasticsearch@googlegroups.com [mailto:
elasticsearch@googlegroups.com] *On Behalf Of *vineeth mohan
Sent: Wednesday, September 10, 2014 5:14 AM
To: elasticsearch@googlegroups.com
Subject: Re: Scripting and dates

Hello Michael ,

Please find the answers in the order of questions you have asked -

Referencing script from file system is explained here. It has very
well worked for me , please double check on it -
Elasticsearch Platform — Find real-time answers at scale | Elastic

I feel you haven't declared that field as date type in the schema .
If you had done that , you will recieve the epoch instead of string. -
Elasticsearch Platform — Find real-time answers at scale | Elastic

Dates are internally stored as epoch. So it should handle that
second fraction too. More on the format can be seen here -
DateTimeFormat (Joda time 2.1 API)

What exactly do you want to do with the duration ? If its range
aggregation , it does have script support -
Elasticsearch Platform — Find real-time answers at scale | Elastic

Thanks
      Vineeth
On Wed, Sep 10, 2014 at 11:32 AM, Michael Giagnocavo mgg@giagnocavo.net
wrote:

I'm trying to work with dates inside a script. I've got a few questions:

How do I reference a script that I have in the scripts directory?
Simply POSTing to /index/type/id/_update with { "script": "scriptname" }
does not seem to work. "No such property: scriptname for class: ScriptN",
where N starts at 3 (I have two .groovy files in my scripts directory).

2: How can I get actual date objects from the source?
ctx._source.fieldname always returns a type string, even if I just created
the field with ctx._source.fieldname = new Date(). Right now I'm parsing
the string output in Groovy, which seems suboptimal.

3: Are ISO8601 dates not fully supported, as far as arbitrary fractional
second decimals? (Not just 3 or another fixed number?) Any suggestions on
handling JSON input from multiple sources, some of which have
high-precision?

4: Can I use a script to project the document into a scalar for
aggregates? For instance, if I have Date fields "start" and "end", and want
to calculate the average duration (start - end) in an aggregate. I see
value-level scripts are allowed, and 1.4 has "scripted metric aggregation".
For now am I best off just storing the duration in the document?

Thank you,
Michael

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/13d4cf783a83447a84b62206605ad312%40CO1PR07MB331.namprd07.prod.outlook.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ef8054ee424145a489ee496117ae0dd4%40CO1PR07MB331.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/ef8054ee424145a489ee496117ae0dd4%40CO1PR07MB331.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5nk9KVGSHZHHx8OUyvGaq6MknOim23%2BBaNnZfw3TEuRDg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5nk9KVGSHZHHx8OUyvGaq6MknOim23%2BBaNnZfw3TEuRDg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2686ebcc594b43049c67fa21419f5555%40CO1PR07MB331.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/2686ebcc594b43049c67fa21419f5555%40CO1PR07MB331.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5nMfEah7-QJpC9bW7n%3Dx0j5FQXwwD6DjGdXPGqU8zrgGg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.