Scripting and dates

Elastic Stack Elasticsearch
1

I'm trying to work with dates inside a script. I've got a few questions:

  1. How do I reference a script that I have in the scripts directory? Simply POSTing to /index/type/id/_update with { "script": "scriptname" } does not seem to work. "No such property: scriptname for class: ScriptN", where N starts at 3 (I have two .groovy files in my scripts directory).

2: How can I get actual date objects from the source? ctx._source.fieldname always returns a type string, even if I just created the field with ctx._source.fieldname = new Date(). Right now I'm parsing the string output in Groovy, which seems suboptimal.

3: Are ISO8601 dates not fully supported, as far as arbitrary fractional second decimals? (Not just 3 or another fixed number?) Any suggestions on handling JSON input from multiple sources, some of which have high-precision?

4: Can I use a script to project the document into a scalar for aggregates? For instance, if I have Date fields "start" and "end", and want to calculate the average duration (start - end) in an aggregate. I see value-level scripts are allowed, and 1.4 has "scripted metric aggregation". For now am I best off just storing the duration in the document?

Thank you,
Michael

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/13d4cf783a83447a84b62206605ad312%40CO1PR07MB331.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

2

Hello Michael ,

Please find the answers in the order of questions you have asked -

  1. Referencing script from file system is explained here. It has very
    well worked for me , please double check on it -
    Elasticsearch Platform — Find real-time answers at scale | Elastic
  2. I feel you haven't declared that field as date type in the schema .
    If you had done that , you will recieve the epoch instead of string. -
    Elasticsearch Platform — Find real-time answers at scale | Elastic
  3. Dates are internally stored as epoch. So it should handle that second
    fraction too. More on the format can be seen here -
    DateTimeFormat (Joda time 2.1 API)
  4. What exactly do you want to do with the duration ? If its range
    aggregation , it does have script support -
    Elasticsearch Platform — Find real-time answers at scale | Elastic

Thanks
Vineeth

On Wed, Sep 10, 2014 at 11:32 AM, Michael Giagnocavo mgg@giagnocavo.net
wrote:

I'm trying to work with dates inside a script. I've got a few questions:

  1. How do I reference a script that I have in the scripts directory?
    Simply POSTing to /index/type/id/_update with { "script": "scriptname" }
    does not seem to work. "No such property: scriptname for class: ScriptN",
    where N starts at 3 (I have two .groovy files in my scripts directory).

2: How can I get actual date objects from the source?
ctx._source.fieldname always returns a type string, even if I just created
the field with ctx._source.fieldname = new Date(). Right now I'm parsing
the string output in Groovy, which seems suboptimal.

3: Are ISO8601 dates not fully supported, as far as arbitrary fractional
second decimals? (Not just 3 or another fixed number?) Any suggestions on
handling JSON input from multiple sources, some of which have
high-precision?

4: Can I use a script to project the document into a scalar for
aggregates? For instance, if I have Date fields "start" and "end", and want
to calculate the average duration (start - end) in an aggregate. I see
value-level scripts are allowed, and 1.4 has "scripted metric aggregation".
For now am I best off just storing the duration in the document?

Thank you,
Michael

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/13d4cf783a83447a84b62206605ad312%40CO1PR07MB331.namprd07.prod.outlook.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Thanks for the reply Vineeth. Here’s a concise example showing the date type problem. Despite having a mapping, the script is getting the properties as strings. How can I verify how ElasticSearch is actually storing something?

$ cat /etc/elasticsearch/scripts/setdur.groovy
ctx._source.dur = (ctx._source.end.getTime() - ctx._source.start.getTime())

$ curl -i -XPUT http://localhost:9200/tesi/ -d '{
"mappings": {
"testt": { "properties": { "start": { "type": "date" }, "end": { "type": "date" } } }
}
}'
HTTP/1.1 200 OK

$ curl -i -XPOST http://localhost:9200/tesi/testt/1 -d '{ "start": "2014-09-01T12:00:00", "end": "2014-09-02T12:00:00" }'
HTTP/1.1 201 Created

rooty@sofab-es1:~$ curl -i -XPOST http://localhost:9200/tesi/testt/1/_update -d '{"script": "setdur", "lang": "groovy"}'
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=UTF-8
Content-Length: 399

{"error":"ElasticsearchIllegalArgumentException[failed to execute script]; nested: GroovyScriptExecutionException[MissingMethodException[No signature of method: java.lang.String.getTime() is applicable for argument types: () values: []\nPossible solutions: getBytes(), trim(), getBytes(java.lang.String), getBytes(java.nio.charset.Charset), getAt(groovy.lang.IntRange), getAt(int)]]; ","status":400}rooty@sofab-es1:~$

Any suggestions?
-Michael

From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of vineeth mohan
Sent: Wednesday, September 10, 2014 5:14 AM
To: elasticsearch@googlegroups.com
Subject: Re: Scripting and dates

Hello Michael ,

Please find the answers in the order of questions you have asked -

  1. Referencing script from file system is explained here. It has very well worked for me , please double check on it - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html
  2. I feel you haven't declared that field as date type in the schema . If you had done that , you will recieve the epoch instead of string. - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date
  3. Dates are internally stored as epoch. So it should handle that second fraction too. More on the format can be seen here - http://joda-time.sourceforge.net/api-release/org/joda/time/format/DateTimeFormat.html
  4. What exactly do you want to do with the duration ? If its range aggregation , it does have script support - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-range-aggregation.html#search-aggregations-bucket-range-aggregation
    Thanks
    Vineeth

On Wed, Sep 10, 2014 at 11:32 AM, Michael Giagnocavo <mgg@giagnocavo.netmailto:mgg@giagnocavo.net> wrote:
I'm trying to work with dates inside a script. I've got a few questions:

  1. How do I reference a script that I have in the scripts directory? Simply POSTing to /index/type/id/_update with { "script": "scriptname" } does not seem to work. "No such property: scriptname for class: ScriptN", where N starts at 3 (I have two .groovy files in my scripts directory).

2: How can I get actual date objects from the source? ctx._source.fieldname always returns a type string, even if I just created the field with ctx._source.fieldname = new Date(). Right now I'm parsing the string output in Groovy, which seems suboptimal.

3: Are ISO8601 dates not fully supported, as far as arbitrary fractional second decimals? (Not just 3 or another fixed number?) Any suggestions on handling JSON input from multiple sources, some of which have high-precision?

4: Can I use a script to project the document into a scalar for aggregates? For instance, if I have Date fields "start" and "end", and want to calculate the average duration (start - end) in an aggregate. I see value-level scripts are allowed, and 1.4 has "scripted metric aggregation". For now am I best off just storing the duration in the document?

Thank you,
Michael

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch%2Bunsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/13d4cf783a83447a84b62206605ad312%40CO1PR07MB331.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/06f511148cb64db49990663de158e9b6%40CO1PR07MB331.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

4

BTW I found the problem with referring to a script by name. If the script has an error, then it fails on compile, written to error log. It’s then not considered a script. ES might want to change that behavior, so if you use “script” : “brokenscript” you get an error indicating what’s actually wrong. Of course if you know about this behavior I guess it’s not a big deal.

From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of vineeth mohan
Sent: Wednesday, September 10, 2014 5:14 AM
To: elasticsearch@googlegroups.com
Subject: Re: Scripting and dates

Hello Michael ,

Please find the answers in the order of questions you have asked -

  1. Referencing script from file system is explained here. It has very well worked for me , please double check on it - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html
  2. I feel you haven't declared that field as date type in the schema . If you had done that , you will recieve the epoch instead of string. - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date
  3. Dates are internally stored as epoch. So it should handle that second fraction too. More on the format can be seen here - http://joda-time.sourceforge.net/api-release/org/joda/time/format/DateTimeFormat.html
  4. What exactly do you want to do with the duration ? If its range aggregation , it does have script support - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-range-aggregation.html#search-aggregations-bucket-range-aggregation
    Thanks
    Vineeth

On Wed, Sep 10, 2014 at 11:32 AM, Michael Giagnocavo <mgg@giagnocavo.netmailto:mgg@giagnocavo.net> wrote:
I'm trying to work with dates inside a script. I've got a few questions:

  1. How do I reference a script that I have in the scripts directory? Simply POSTing to /index/type/id/_update with { "script": "scriptname" } does not seem to work. "No such property: scriptname for class: ScriptN", where N starts at 3 (I have two .groovy files in my scripts directory).

2: How can I get actual date objects from the source? ctx._source.fieldname always returns a type string, even if I just created the field with ctx._source.fieldname = new Date(). Right now I'm parsing the string output in Groovy, which seems suboptimal.

3: Are ISO8601 dates not fully supported, as far as arbitrary fractional second decimals? (Not just 3 or another fixed number?) Any suggestions on handling JSON input from multiple sources, some of which have high-precision?

4: Can I use a script to project the document into a scalar for aggregates? For instance, if I have Date fields "start" and "end", and want to calculate the average duration (start - end) in an aggregate. I see value-level scripts are allowed, and 1.4 has "scripted metric aggregation". For now am I best off just storing the duration in the document?

Thank you,
Michael

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch%2Bunsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/13d4cf783a83447a84b62206605ad312%40CO1PR07MB331.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ef8054ee424145a489ee496117ae0dd4%40CO1PR07MB331.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

5

Hello Michael ,

This should work for you -

cat scr
{
"script": "sdf = new
java.text.SimpleDateFormat('yyyy-MM-dd\'T\'HH:mm:ss');startDate =
sdf.parse(ctx._source.start);endDate = sdf.parse(ctx._source.end);
ctx._source.diff = endDate.getTime() - startDate.getTime();"
}
curl -XPOST http://localhost:9200/test/logs/1/_update -d @scr
{"_index":"test","_type":"logs","_id":"1","_version":10

I was in the pretext that one can access the doc object for updation , but
seems we can only access _source here.
In that case , you need to parse the string to date object and then do the
stuffs.
The above works for me perfectly.

Thanks
Vineeth

On Thu, Sep 11, 2014 at 12:12 AM, Michael Giagnocavo mgg@giagnocavo.net
wrote:

BTW I found the problem with referring to a script by name. If the
script has an error, then it fails on compile, written to error log. It’s
then not considered a script. ES might want to change that behavior, so if
you use “script” : “brokenscript” you get an error indicating what’s
actually wrong. Of course if you know about this behavior I guess it’s not
a big deal.

From: elasticsearch@googlegroups.com [mailto:
elasticsearch@googlegroups.com] *On Behalf Of *vineeth mohan
Sent: Wednesday, September 10, 2014 5:14 AM
To: elasticsearch@googlegroups.com
Subject: Re: Scripting and dates

Hello Michael ,

Please find the answers in the order of questions you have asked -

  1. Referencing script from file system is explained here. It has very
    well worked for me , please double check on it -
    Elasticsearch Platform — Find real-time answers at scale | Elastic
  2. I feel you haven't declared that field as date type in the schema .
    If you had done that , you will recieve the epoch instead of string. -
    Elasticsearch Platform — Find real-time answers at scale | Elastic
  3. Dates are internally stored as epoch. So it should handle that
    second fraction too. More on the format can be seen here -
    DateTimeFormat (Joda time 2.1 API)
  4. What exactly do you want to do with the duration ? If its range
    aggregation , it does have script support -
    Elasticsearch Platform — Find real-time answers at scale | Elastic

Thanks

      Vineeth

On Wed, Sep 10, 2014 at 11:32 AM, Michael Giagnocavo mgg@giagnocavo.net
wrote:

I'm trying to work with dates inside a script. I've got a few questions:

  1. How do I reference a script that I have in the scripts directory?
    Simply POSTing to /index/type/id/_update with { "script": "scriptname" }
    does not seem to work. "No such property: scriptname for class: ScriptN",
    where N starts at 3 (I have two .groovy files in my scripts directory).

2: How can I get actual date objects from the source?
ctx._source.fieldname always returns a type string, even if I just created
the field with ctx._source.fieldname = new Date(). Right now I'm parsing
the string output in Groovy, which seems suboptimal.

3: Are ISO8601 dates not fully supported, as far as arbitrary fractional
second decimals? (Not just 3 or another fixed number?) Any suggestions on
handling JSON input from multiple sources, some of which have
high-precision?

4: Can I use a script to project the document into a scalar for
aggregates? For instance, if I have Date fields "start" and "end", and want
to calculate the average duration (start - end) in an aggregate. I see
value-level scripts are allowed, and 1.4 has "scripted metric aggregation".
For now am I best off just storing the duration in the document?

Thank you,
Michael

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/13d4cf783a83447a84b62206605ad312%40CO1PR07MB331.namprd07.prod.outlook.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ef8054ee424145a489ee496117ae0dd4%40CO1PR07MB331.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/ef8054ee424145a489ee496117ae0dd4%40CO1PR07MB331.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5nk9KVGSHZHHx8OUyvGaq6MknOim23%2BBaNnZfw3TEuRDg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

6

Thank you, I did have to resort to parsing the date string. The next problem is that dates don’t seem to come back in a fixed format. If I post a date with just the date string, that’s what I get back. Likewise for fractional seconds. So I do some string work with substring first.

How can I determine if ES has actually indexed a value as a date?

-Michael

From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of vineeth mohan
Sent: Wednesday, September 10, 2014 2:13 PM
To: elasticsearch@googlegroups.com
Subject: Re: Scripting and dates

Hello Michael ,

This should work for you -

cat scr
{
"script": "sdf = new java.text.SimpleDateFormat('yyyy-MM-dd\'T\'HH:mm:ss');startDate = sdf.parse(ctx._source.start);endDate = sdf.parse(ctx._source.end); ctx._source.diff = endDate.getTime() - startDate.getTime();"
}
curl -XPOST http://localhost:9200/test/logs/1/_update -d @scr
{"_index":"test","_type":"logs","_id":"1","_version":10

I was in the pretext that one can access the doc object for updation , but seems we can only access _source here.
In that case , you need to parse the string to date object and then do the stuffs.
The above works for me perfectly.

Thanks
Vineeth

On Thu, Sep 11, 2014 at 12:12 AM, Michael Giagnocavo <mgg@giagnocavo.netmailto:mgg@giagnocavo.net> wrote:
BTW I found the problem with referring to a script by name. If the script has an error, then it fails on compile, written to error log. It’s then not considered a script. ES might want to change that behavior, so if you use “script” : “brokenscript” you get an error indicating what’s actually wrong. Of course if you know about this behavior I guess it’s not a big deal.

From: elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com] On Behalf Of vineeth mohan
Sent: Wednesday, September 10, 2014 5:14 AM
To: elasticsearch@googlegroups.commailto:elasticsearch@googlegroups.com
Subject: Re: Scripting and dates

Hello Michael ,

Please find the answers in the order of questions you have asked -

  1. Referencing script from file system is explained here. It has very well worked for me , please double check on it - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html
  2. I feel you haven't declared that field as date type in the schema . If you had done that , you will recieve the epoch instead of string. - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#date
  3. Dates are internally stored as epoch. So it should handle that second fraction too. More on the format can be seen here - http://joda-time.sourceforge.net/api-release/org/joda/time/format/DateTimeFormat.html
  4. What exactly do you want to do with the duration ? If its range aggregation , it does have script support - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-range-aggregation.html#search-aggregations-bucket-range-aggregation
    Thanks
    Vineeth

On Wed, Sep 10, 2014 at 11:32 AM, Michael Giagnocavo <mgg@giagnocavo.netmailto:mgg@giagnocavo.net> wrote:
I'm trying to work with dates inside a script. I've got a few questions:

  1. How do I reference a script that I have in the scripts directory? Simply POSTing to /index/type/id/_update with { "script": "scriptname" } does not seem to work. "No such property: scriptname for class: ScriptN", where N starts at 3 (I have two .groovy files in my scripts directory).

2: How can I get actual date objects from the source? ctx._source.fieldname always returns a type string, even if I just created the field with ctx._source.fieldname = new Date(). Right now I'm parsing the string output in Groovy, which seems suboptimal.

3: Are ISO8601 dates not fully supported, as far as arbitrary fractional second decimals? (Not just 3 or another fixed number?) Any suggestions on handling JSON input from multiple sources, some of which have high-precision?

4: Can I use a script to project the document into a scalar for aggregates? For instance, if I have Date fields "start" and "end", and want to calculate the average duration (start - end) in an aggregate. I see value-level scripts are allowed, and 1.4 has "scripted metric aggregation". For now am I best off just storing the duration in the document?

Thank you,
Michael

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch%2Bunsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/13d4cf783a83447a84b62206605ad312%40CO1PR07MB331.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ef8054ee424145a489ee496117ae0dd4%40CO1PR07MB331.namprd07.prod.outlook.comhttps://groups.google.com/d/msgid/elasticsearch/ef8054ee424145a489ee496117ae0dd4%40CO1PR07MB331.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5nk9KVGSHZHHx8OUyvGaq6MknOim23%2BBaNnZfw3TEuRDg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAGdPd5nk9KVGSHZHHx8OUyvGaq6MknOim23%2BBaNnZfw3TEuRDg%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2686ebcc594b43049c67fa21419f5555%40CO1PR07MB331.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

7

Hello Michael ,

When you set the type of the field as time , its indexed as time ( as
epoch to be more precise ).
And by default , it takes multiple date formats.
But the issue here is that , the index data is not available to you while
updating.
Only the stored data which is the date string is available.

So there are 2 options here

  1. While declaring a field as type date , you can also specify the
    format in which the date string comes in. Any other format is rejected.
    This way hard coding the format in the script should work fine.
  2. Store the epoch representation as time instead of date string. This
    wont in anyway hinter with any of date operation like bucketing ,range
    selection but at the same time , your update operations would be smooth.

Thanks
Vineeth

On Thu, Sep 11, 2014 at 5:16 AM, Michael Giagnocavo mgg@giagnocavo.net
wrote:

Thank you, I did have to resort to parsing the date string. The next
problem is that dates don’t seem to come back in a fixed format. If I post
a date with just the date string, that’s what I get back. Likewise for
fractional seconds. So I do some string work with substring first.

How can I determine if ES has actually indexed a value as a date?

-Michael

From: elasticsearch@googlegroups.com [mailto:
elasticsearch@googlegroups.com] *On Behalf Of *vineeth mohan
Sent: Wednesday, September 10, 2014 2:13 PM

To: elasticsearch@googlegroups.com
Subject: Re: Scripting and dates

Hello Michael ,

This should work for you -

cat scr

{

        "script": "sdf = new

java.text.SimpleDateFormat('yyyy-MM-dd\'T\'HH:mm:ss');startDate =
sdf.parse(ctx._source.start);endDate = sdf.parse(ctx._source.end);
ctx._source.diff = endDate.getTime() - startDate.getTime();"

}

curl -XPOST http://localhost:9200/test/logs/1/_update -d @scr

{"_index":"test","_type":"logs","_id":"1","_version":10

I was in the pretext that one can access the doc object for updation , but
seems we can only access _source here.

In that case , you need to parse the string to date object and then do the
stuffs.

The above works for me perfectly.

Thanks

       Vineeth

On Thu, Sep 11, 2014 at 12:12 AM, Michael Giagnocavo mgg@giagnocavo.net
wrote:

BTW I found the problem with referring to a script by name. If the
script has an error, then it fails on compile, written to error log. It’s
then not considered a script. ES might want to change that behavior, so if
you use “script” : “brokenscript” you get an error indicating what’s
actually wrong. Of course if you know about this behavior I guess it’s not
a big deal.

From: elasticsearch@googlegroups.com [mailto:
elasticsearch@googlegroups.com] *On Behalf Of *vineeth mohan
Sent: Wednesday, September 10, 2014 5:14 AM
To: elasticsearch@googlegroups.com
Subject: Re: Scripting and dates

Hello Michael ,

Please find the answers in the order of questions you have asked -

  1. Referencing script from file system is explained here. It has very
    well worked for me , please double check on it -
    Elasticsearch Platform — Find real-time answers at scale | Elastic
  2. I feel you haven't declared that field as date type in the schema .
    If you had done that , you will recieve the epoch instead of string. -
    Elasticsearch Platform — Find real-time answers at scale | Elastic
  3. Dates are internally stored as epoch. So it should handle that
    second fraction too. More on the format can be seen here -
    DateTimeFormat (Joda time 2.1 API)
  4. What exactly do you want to do with the duration ? If its range
    aggregation , it does have script support -
    Elasticsearch Platform — Find real-time answers at scale | Elastic

Thanks

      Vineeth

On Wed, Sep 10, 2014 at 11:32 AM, Michael Giagnocavo mgg@giagnocavo.net
wrote:

I'm trying to work with dates inside a script. I've got a few questions:

  1. How do I reference a script that I have in the scripts directory?
    Simply POSTing to /index/type/id/_update with { "script": "scriptname" }
    does not seem to work. "No such property: scriptname for class: ScriptN",
    where N starts at 3 (I have two .groovy files in my scripts directory).

2: How can I get actual date objects from the source?
ctx._source.fieldname always returns a type string, even if I just created
the field with ctx._source.fieldname = new Date(). Right now I'm parsing
the string output in Groovy, which seems suboptimal.

3: Are ISO8601 dates not fully supported, as far as arbitrary fractional
second decimals? (Not just 3 or another fixed number?) Any suggestions on
handling JSON input from multiple sources, some of which have
high-precision?

4: Can I use a script to project the document into a scalar for
aggregates? For instance, if I have Date fields "start" and "end", and want
to calculate the average duration (start - end) in an aggregate. I see
value-level scripts are allowed, and 1.4 has "scripted metric aggregation".
For now am I best off just storing the duration in the document?

Thank you,
Michael

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/13d4cf783a83447a84b62206605ad312%40CO1PR07MB331.namprd07.prod.outlook.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DBzthM14yz3SuzxvTz5QXOW4Gtt72rvsA1-dND5eP--A%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ef8054ee424145a489ee496117ae0dd4%40CO1PR07MB331.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/ef8054ee424145a489ee496117ae0dd4%40CO1PR07MB331.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5nk9KVGSHZHHx8OUyvGaq6MknOim23%2BBaNnZfw3TEuRDg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5nk9KVGSHZHHx8OUyvGaq6MknOim23%2BBaNnZfw3TEuRDg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2686ebcc594b43049c67fa21419f5555%40CO1PR07MB331.namprd07.prod.outlook.com
https://groups.google.com/d/msgid/elasticsearch/2686ebcc594b43049c67fa21419f5555%40CO1PR07MB331.namprd07.prod.outlook.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5nMfEah7-QJpC9bW7n%3Dx0j5FQXwwD6DjGdXPGqU8zrgGg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.