Timestamp

Hello,

I'm trying to use the timestamp feature to automatically add a timestamp to
the newly indexed doc.
Since we are already indexing document without timestamp I figured out that
I could simply execute:

curl -XPUT 'http://localhost:9200/ubilogs-mbr/_mapping/logs' -d '{
"logs" : {
"_timestamp" : {
"enabled" : true,
"stored" : true,
"format": "YYYY-MM-dd HH:mm:ss.SSS"
}
}
}'

and the next document to be indexed will have the _timestamp field added.
But when I search for the doc with, for example:
{"fields":["_timestamp"],"query":{"match_all":{}}}

I only get

{

  • "_index": "ubilogs-mbr",
  • "_type": "logs",
  • "_id": "AUyVGpmfoHcL_d0dTPU4",
  • "_score": 1

}

no _timestamp field here.
Is there anything else to do?

Is my understanding correct: ES cluster will automatically generate a
default _timestamp with the time when the doc is indexed?

Antoine

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6a82defe-a17e-4b41-977a-584da43a41ad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Are you doing that using curl or any interface within your browser?

If the later, can you try again with curl or Marvel/Sense.

If it does not work, gist a full script.

BTW what is your version?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 7 avr. 2015 à 18:20, Antoine Brun abr@ubiqube.com a écrit :

Hello,

I'm trying to use the timestamp feature to automatically add a timestamp to the newly indexed doc.
Since we are already indexing document without timestamp I figured out that I could simply execute:

curl -XPUT 'http://localhost:9200/ubilogs-mbr/_mapping/logs' -d '{
"logs" : {
"_timestamp" : {
"enabled" : true,
"stored" : true,
"format": "YYYY-MM-dd HH:mm:ss.SSS"
}
}
}'

and the next document to be indexed will have the _timestamp field added.
But when I search for the doc with, for example:
{"fields":["_timestamp"],"query":{"match_all":{}}}

I only get

{
"_index": "ubilogs-mbr",
"_type": "logs",
"_id": "AUyVGpmfoHcL_d0dTPU4",
"_score": 1
}

no _timestamp field here.
Is there anything else to do?

Is my understanding correct: ES cluster will automatically generate a default _timestamp with the time when the doc is indexed?

Antoine

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6a82defe-a17e-4b41-977a-584da43a41ad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/423AA837-8789-4123-BE16-7A130ED183A1%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Hello,

I'm using curl directly from the Linux CLI.
The configuration is correctly applied because when I do
curl -XGET http://localhost:9200/ubilogs-mbr/_mapping

I get this:

{
"ubilogs-mbr": {
"mappings": {
"logs": {
"dynamic": "strict",

"_timestamp": { "enabled": true,
"format": "yyyy-MM-dd HH:mm:ss" },

"properties": {
"action": {
"type": "string",
"index": "not_analyzed",
"store": true,
"doc_values": true
},
"application": {
"type": "string",
"index": "not_analyzed",
"store": true,
"doc_values": true
},
"attack": {
"type": "string",
"index": "not_analyzed",
"store": true,
"doc_values": true
},
...
}
}
}
}
}

On Tue, Apr 7, 2015 at 7:24 PM, David Pilato david@pilato.fr wrote:

Are you doing that using curl or any interface within your browser?

If the later, can you try again with curl or Marvel/Sense.

If it does not work, gist a full script.

BTW what is your version?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 7 avr. 2015 à 18:20, Antoine Brun abr@ubiqube.com a écrit :

Hello,

I'm trying to use the timestamp feature to automatically add a timestamp
to the newly indexed doc.
Since we are already indexing document without timestamp I figured out
that I could simply execute:

curl -XPUT 'http://localhost:9200/ubilogs-mbr/_mapping/logs' -d '{
"logs" : {
"_timestamp" : {
"enabled" : true,
"stored" : true,
"format": "YYYY-MM-dd HH:mm:ss.SSS"
}
}
}'

and the next document to be indexed will have the _timestamp field added.
But when I search for the doc with, for example:
{"fields":["_timestamp"],"query":{"match_all":{}}}

I only get

{

  • "_index": "ubilogs-mbr",
  • "_type": "logs",
  • "_id": "AUyVGpmfoHcL_d0dTPU4",
  • "_score": 1

}

no _timestamp field here.
Is there anything else to do?

Is my understanding correct: ES cluster will automatically generate a
default _timestamp with the time when the doc is indexed?

Antoine

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6a82defe-a17e-4b41-977a-584da43a41ad%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/6a82defe-a17e-4b41-977a-584da43a41ad%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/JX95IGNBjt0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/423AA837-8789-4123-BE16-7A130ED183A1%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/423AA837-8789-4123-BE16-7A130ED183A1%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Antoine

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGUV-q-qzr3S28tBAxjF3nxEzkTgJdOQeF3SsYGPRbeJeP0%3D1A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

So _timestamp is not stored.
That’s your concern here.

--
David Pilato - Developer | Evangelist
elastic.co
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs

Le 7 avr. 2015 à 20:41, Antoine Brun antoine.brun@gmail.com a écrit :

Hello,

I'm using curl directly from the Linux CLI.
The configuration is correctly applied because when I do
curl -XGET http://localhost:9200/ubilogs-mbr/_mapping http://localhost:9200/ubilogs-mbr/_mapping

I get this:

{
"ubilogs-mbr": {
"mappings": {
"logs": {
"dynamic": "strict",
"_timestamp": {
"enabled": true,
"format": "yyyy-MM-dd HH:mm:ss"
},
"properties": {
"action": {
"type": "string",
"index": "not_analyzed",
"store": true,
"doc_values": true
},
"application": {
"type": "string",
"index": "not_analyzed",
"store": true,
"doc_values": true
},
"attack": {
"type": "string",
"index": "not_analyzed",
"store": true,
"doc_values": true
},
...
}
}
}
}
}

On Tue, Apr 7, 2015 at 7:24 PM, David Pilato <david@pilato.fr mailto:david@pilato.fr> wrote:
Are you doing that using curl or any interface within your browser?

If the later, can you try again with curl or Marvel/Sense.

If it does not work, gist a full script.

BTW what is your version?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 7 avr. 2015 à 18:20, Antoine Brun <abr@ubiqube.com mailto:abr@ubiqube.com> a écrit :

Hello,

I'm trying to use the timestamp feature to automatically add a timestamp to the newly indexed doc.
Since we are already indexing document without timestamp I figured out that I could simply execute:

curl -XPUT 'http://localhost:9200/ubilogs-mbr/_mapping/logs' http://localhost:9200/ubilogs-mbr/_mapping/logs' -d '{
"logs" : {
"_timestamp" : {
"enabled" : true,
"stored" : true,
"format": "YYYY-MM-dd HH:mm:ss.SSS"
}
}
}'

and the next document to be indexed will have the _timestamp field added.
But when I search for the doc with, for example:
{"fields":["_timestamp"],"query":{"match_all":{}}}

I only get

{
"_index": "ubilogs-mbr",
"_type": "logs",
"_id": "AUyVGpmfoHcL_d0dTPU4",
"_score": 1
}

no _timestamp field here.
Is there anything else to do?

Is my understanding correct: ES cluster will automatically generate a default _timestamp with the time when the doc is indexed?

Antoine

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6a82defe-a17e-4b41-977a-584da43a41ad%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/6a82defe-a17e-4b41-977a-584da43a41ad%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/JX95IGNBjt0/unsubscribe https://groups.google.com/d/topic/elasticsearch/JX95IGNBjt0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/423AA837-8789-4123-BE16-7A130ED183A1%40pilato.fr https://groups.google.com/d/msgid/elasticsearch/423AA837-8789-4123-BE16-7A130ED183A1%40pilato.fr?utm_medium=email&utm_source=footer.

For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
Antoine

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGUV-q-qzr3S28tBAxjF3nxEzkTgJdOQeF3SsYGPRbeJeP0%3D1A%40mail.gmail.com https://groups.google.com/d/msgid/elasticsearch/CAGUV-q-qzr3S28tBAxjF3nxEzkTgJdOQeF3SsYGPRbeJeP0%3D1A%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/EBF879C3-13A4-47A1-845E-CD4DF28D16B2%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Hello,

I solved my problem, thanks.
actually the rest command should be:

curl -XPUT 'http://localhost:9200/ubilogs-mbr/_mapping/logs' -d '{
"logs" : {
"_timestamp" : {
"enabled" : true,
"store" : true,
"format": "YYYY-MM-dd HH:mm:ss.SSS"
}
}
}'

with "store" and not "stored", my mistake.
Thanks for your answers

Antoine

On Tue, Apr 7, 2015 at 8:45 PM, David Pilato david@pilato.fr wrote:

So _timestamp is not stored.
That’s your concern here.

--
David Pilato - Developer | Evangelist
elastic.co http://elastic.co
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 7 avr. 2015 à 20:41, Antoine Brun antoine.brun@gmail.com a écrit :

Hello,

I'm using curl directly from the Linux CLI.
The configuration is correctly applied because when I do
curl -XGET http://localhost:9200/ubilogs-mbr/_mapping

I get this:

{
"ubilogs-mbr": {
"mappings": {
"logs": {
"dynamic": "strict",

"_timestamp": { "enabled": true,
"format": "yyyy-MM-dd HH:mm:ss" },

"properties": {
"action": {
"type": "string",
"index": "not_analyzed",
"store": true,
"doc_values": true
},
"application": {
"type": "string",
"index": "not_analyzed",
"store": true,
"doc_values": true
},
"attack": {
"type": "string",
"index": "not_analyzed",
"store": true,
"doc_values": true
},
...
}
}
}
}
}

On Tue, Apr 7, 2015 at 7:24 PM, David Pilato david@pilato.fr wrote:

Are you doing that using curl or any interface within your browser?

If the later, can you try again with curl or Marvel/Sense.

If it does not work, gist a full script.

BTW what is your version?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 7 avr. 2015 à 18:20, Antoine Brun abr@ubiqube.com a écrit :

Hello,

I'm trying to use the timestamp feature to automatically add a timestamp
to the newly indexed doc.
Since we are already indexing document without timestamp I figured out
that I could simply execute:

curl -XPUT 'http://localhost:9200/ubilogs-mbr/_mapping/logs' -d '{
"logs" : {
"_timestamp" : {
"enabled" : true,
"stored" : true,
"format": "YYYY-MM-dd HH:mm:ss.SSS"
}
}
}'

and the next document to be indexed will have the _timestamp field added.
But when I search for the doc with, for example:
{"fields":["_timestamp"],"query":{"match_all":{}}}

I only get

{

  • "_index": "ubilogs-mbr",
  • "_type": "logs",
  • "_id": "AUyVGpmfoHcL_d0dTPU4",
  • "_score": 1

}

no _timestamp field here.
Is there anything else to do?

Is my understanding correct: ES cluster will automatically generate a
default _timestamp with the time when the doc is indexed?

Antoine

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6a82defe-a17e-4b41-977a-584da43a41ad%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/6a82defe-a17e-4b41-977a-584da43a41ad%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/JX95IGNBjt0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/423AA837-8789-4123-BE16-7A130ED183A1%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/423AA837-8789-4123-BE16-7A130ED183A1%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Antoine

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGUV-q-qzr3S28tBAxjF3nxEzkTgJdOQeF3SsYGPRbeJeP0%3D1A%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAGUV-q-qzr3S28tBAxjF3nxEzkTgJdOQeF3SsYGPRbeJeP0%3D1A%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/JX95IGNBjt0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/EBF879C3-13A4-47A1-845E-CD4DF28D16B2%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/EBF879C3-13A4-47A1-845E-CD4DF28D16B2%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Antoine

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGUV-q8ZiQ4rEGCCosUk5M0QT-WFhGiUZFZ1bHRa4wqXTFtM3w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Hello,

based on the comments I could create a new index with _timestamp activated
and it works great.
Now my probleme arrives when I want to activate the timestamp on an
existing index.
Since _timestamp is not stored by default, I wanted to set "store" to true,
but I get

{
"error": "MergeMappingException[Merge failed with failures {[mapper
[_timestamp] has different store values]}]",
"status": 400
}

Is there any way to achieve this?

Antoine

Le mardi 7 avril 2015 18:20:33 UTC+2, Antoine Brun a écrit :

Hello,

I'm trying to use the timestamp feature to automatically add a timestamp
to the newly indexed doc.
Since we are already indexing document without timestamp I figured out
that I could simply execute:

curl -XPUT 'http://localhost:9200/ubilogs-mbr/_mapping/logs' -d '{
"logs" : {
"_timestamp" : {
"enabled" : true,
"stored" : true,
"format": "YYYY-MM-dd HH:mm:ss.SSS"
}
}
}'

and the next document to be indexed will have the _timestamp field added.
But when I search for the doc with, for example:
{"fields":["_timestamp"],"query":{"match_all":{}}}

I only get

{

  • "_index": "ubilogs-mbr",
  • "_type": "logs",
  • "_id": "AUyVGpmfoHcL_d0dTPU4",
  • "_score": 1

}

no _timestamp field here.
Is there anything else to do?

Is my understanding correct: ES cluster will automatically generate a
default _timestamp with the time when the doc is indexed?

Antoine

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c2cad5d1-038c-4a76-a9e8-1dca1ea81d59%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You need to reindex in a new index.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 16 avr. 2015 à 17:33, Antoine Brun abr@ubiqube.com a écrit :

Hello,

based on the comments I could create a new index with _timestamp activated and it works great.
Now my probleme arrives when I want to activate the timestamp on an existing index.
Since _timestamp is not stored by default, I wanted to set "store" to true, but I get

{
"error": "MergeMappingException[Merge failed with failures {[mapper [_timestamp] has different store values]}]",
"status": 400
}

Is there any way to achieve this?

Antoine

Le mardi 7 avril 2015 18:20:33 UTC+2, Antoine Brun a écrit :

Hello,

I'm trying to use the timestamp feature to automatically add a timestamp to the newly indexed doc.
Since we are already indexing document without timestamp I figured out that I could simply execute:

curl -XPUT 'http://localhost:9200/ubilogs-mbr/_mapping/logs' -d '{
"logs" : {
"_timestamp" : {
"enabled" : true,
"stored" : true,
"format": "YYYY-MM-dd HH:mm:ss.SSS"
}
}
}'

and the next document to be indexed will have the _timestamp field added.
But when I search for the doc with, for example:
{"fields":["_timestamp"],"query":{"match_all":{}}}

I only get

{
"_index": "ubilogs-mbr",
"_type": "logs",
"_id": "AUyVGpmfoHcL_d0dTPU4",
"_score": 1
}

no _timestamp field here.
Is there anything else to do?

Is my understanding correct: ES cluster will automatically generate a default _timestamp with the time when the doc is indexed?

Antoine

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c2cad5d1-038c-4a76-a9e8-1dca1ea81d59%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/E558667C-C0B1-4AAD-85D0-956D51F9D2B8%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.