No _timestamp possible

Bernd_Fehling · October 2, 2013, 1:45pm

I tried many different options/combinations to get a _timestamp of the
current time (like "now")
set for each document loaded but have no luck to manage this.
As far as I understood from the docs it should be possible with:
{
"mappings" : {
"docs" : {
"_timestamp" : { "enabled" : "yes", "store" : "yes" },
"properties" : {
"id" : { "type" : "string", "index" : "not_analyzed", "store" :
"yes" },
...
}
}
}
}

Is it true that I should get a _timestamp for each hit in the result?
I'm seeing nothing
Only _index, _type, _id, _score and _source.

Any idea whats wrong?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · October 2, 2013, 2:18pm

Here is an example, the fields to be returned except _source have to be
declared by "fields".

gist.github.com

https://gist.github.com/jprante/6794460

timestamp.sh

curl -XDELETE 'localhost:9200/test'

curl -XPUT 'localhost:9200/test/' -d '
{
    "mappings" : {
        "_default_": {
            "_timestamp" : { "enabled" : true, "store" : true }
        }
    }
}

This file has been truncated. show original

http://www.elasticsearch.org/guide/reference/api/search/fields/

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

brian_yoder · October 2, 2013, 3:00pm

Jörg,

In previous versions of ES, I noticed that when I wanted the _ttl field to
be returned (time to live), I needed to add _ttl to the list of fields. But
then, I also needed to explicitly add _source to the list of fields to
return.

Is this still the case? The documentation seems to imply that _source is
always returned by default even when other fields are asked for.

Thanks!

Brian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · October 2, 2013, 3:09pm

Yes, if there is no "fields", _source is returned by default, otherwise,
you have to declare _source together with your favorite fields.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Thomas_Bolis · October 3, 2013, 1:25pm

Hi,

You have to explicitely request for the particular field "_timestamp" if
you store it in the index only, otherwise you can only do queries upon it
if it is enabled as you do

T.

On Wednesday, 2 October 2013 16:45:19 UTC+3, Bernd Fehling wrote:

I tried many different options/combinations to get a _timestamp of the
current time (like "now")
set for each document loaded but have no luck to manage this.
As far as I understood from the docs it should be possible with:
{
"mappings" : {
"docs" : {
"_timestamp" : { "enabled" : "yes", "store" : "yes" },
"properties" : {
"id" : { "type" : "string", "index" : "not_analyzed", "store" :
"yes" },
...
}
}
}
}

Is it true that I should get a _timestamp for each hit in the result?
I'm seeing nothing
Only _index, _type, _id, _score and _source.

Any idea whats wrong?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Bernd_Fehling · October 8, 2013, 10:24am

OK, now I get/see the _timestamp field, but how to get a timestamp within
_source?
I tried:
{
"mappings" : {
"docs" : {
"properties" : {
"timestamp" : { "type" : "date", "format" : "date_time", "index" :
"not_analyzed", "store" : "true", "null_value" : "now" },
"id" : { "type" : "string", "index" : "not_analyzed", "store" :
"yes" },
...
}
}
}
}

My assumption is, that a field "timestamp" of type "date" with
default_value "now" will be generated?
And it should set the current timestamp (now) if input from "timestamp" is
""?
And I should see it in _source?

Or do I have to send "tmestamp" : "null" as default from my data?

Very difficult to understand this timestamp.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · October 8, 2013, 11:10am

Unfortunately, "now" works only at query time. Also, date is not a core
type which can carry a default value. So for external dates, they have to
be given explicitly in the _source. ES does not modify _source when
indexing. The auto timestamping is only available for the internal
_timestamp field.

Jörg
Am 08.10.2013 12:24 schrieb "Bernd Fehling" bernd.fehling@googlemail.com:

OK, now I get/see the _timestamp field, but how to get a timestamp within
_source?
I tried:
{
"mappings" : {
"docs" : {
"properties" : {
"timestamp" : { "type" : "date", "format" : "date_time", "index"
: "not_analyzed", "store" : "true", "null_value" : "now" },
"id" : { "type" : "string", "index" : "not_analyzed", "store" :
"yes" },
...
}
}
}
}

My assumption is, that a field "timestamp" of type "date" with
default_value "now" will be generated?
And it should set the current timestamp (now) if input from "timestamp" is
""?
And I should see it in _source?

Or do I have to send "tmestamp" : "null" as default from my data?

Very difficult to understand this timestamp.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Bernd_Fehling · October 8, 2013, 11:36am

Thats bad. So I have to stick with SOLR
Everything else could be converted (including Thesaurus). Only one self
written ES-plugin required.
Why is ES not having a "type" : "date" field with "default_value" : "now"?

Can't belief that I'm the only one using timestamps to delete old records
from the index after updating.
Is it possible to enhance ES with a new field type via plugin?

Bernd

Am Dienstag, 8. Oktober 2013 13:10:33 UTC+2 schrieb Jörg Prante:

Unfortunately, "now" works only at query time. Also, date is not a core
type which can carry a default value. So for external dates, they have to
be given explicitly in the _source. ES does not modify _source when
indexing. The auto timestamping is only available for the internal
_timestamp field.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · October 8, 2013, 11:45am

I do use timestamps, also for my (upcoming) OAI ES plugin.

Not sure about the challenge you are facing. You can use _timestamp field
if you want auto timestamping.

Jörg
Am 08.10.2013 13:36 schrieb "Bernd Fehling" bernd.fehling@googlemail.com:

Thats bad. So I have to stick with SOLR
Everything else could be converted (including Thesaurus). Only one self
written ES-plugin required.
Why is ES not having a "type" : "date" field with "default_value" : "now"?

Can't belief that I'm the only one using timestamps to delete old records
from the index after updating.
Is it possible to enhance ES with a new field type via plugin?

Bernd

Am Dienstag, 8. Oktober 2013 13:10:33 UTC+2 schrieb Jörg Prante:

Unfortunately, "now" works only at query time. Also, date is not a core
type which can carry a default value. So for external dates, they have to
be given explicitly in the _source. ES does not modify _source when
indexing. The auto timestamping is only available for the internal
_timestamp field.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Bernd_Fehling · October 8, 2013, 12:15pm

Yes, but first it is not in _source.
Second, my query is: ... "fields": [ "_timestamp", "_source" ] , but I get
_index: base,
_type: docs,
_id: myid1,
_score: 1,
_source: {...},
fields: { _timestamp: 1381226793870 }

And I assume if I query for "fields": [ "_timestamp", "_source" ] to get
_index: base,
_type: docs,
_id: myid1,
_score: 1,
fields: { _timestamp: 1381226793870, _source: {...} }

This is inconsistent, _source is not within fields.
I have to reparse all this crap to get either _timestamp into _source or
_source into fields.
Any other idea for configuration?

Bernd

Am Dienstag, 8. Oktober 2013 13:45:12 UTC+2 schrieb Jörg Prante:

I do use timestamps, also for my (upcoming) OAI ES plugin.

Not sure about the challenge you are facing. You can use _timestamp field
if you want auto timestamping.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · October 8, 2013, 1:43pm

My solution was adding timestamps as ISO dates in the _source, and pointing
the internal _timestamp to it.

Jörg
Am 08.10.2013 14:15 schrieb "Bernd Fehling" bernd.fehling@googlemail.com:

Yes, but first it is not in _source.
Second, my query is: ... "fields": [ "_timestamp", "_source" ] , but I get
_index: base,
_type: docs,
_id: myid1,
_score: 1,
_source: {...},
fields: { _timestamp: 1381226793870 }

And I assume if I query for "fields": [ "_timestamp", "_source" ] to get
_index: base,
_type: docs,
_id: myid1,
_score: 1,
fields: { _timestamp: 1381226793870, _source: {...} }

This is inconsistent, _source is not within fields.
I have to reparse all this crap to get either _timestamp into _source or
_source into fields.
Any other idea for configuration?

Bernd

Am Dienstag, 8. Oktober 2013 13:45:12 UTC+2 schrieb Jörg Prante:

I do use timestamps, also for my (upcoming) OAI ES plugin.

Not sure about the challenge you are facing. You can use _timestamp field
if you want auto timestamping.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Bernd_Fehling · October 11, 2013, 9:22am

How can I point the internal _timestamp to a _source field?

Bernd

Am Dienstag, 8. Oktober 2013 15:43:38 UTC+2 schrieb Jörg Prante:

My solution was adding timestamps as ISO dates in the _source, and
pointing the internal _timestamp to it.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · October 11, 2013, 11:25am

There is the "path" setting

Jörg
Am 11.10.2013 11:22 schrieb "Bernd Fehling" bernd.fehling@googlemail.com:

How can I point the internal _timestamp to a _source field?

Bernd

Am Dienstag, 8. Oktober 2013 15:43:38 UTC+2 schrieb Jörg Prante:

My solution was adding timestamps as ISO dates in the _source, and
pointing the internal _timestamp to it.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.