How does Elasticsearch convert dates to JSON string representations?

Hello everyone,

I have a question on how Elasticsearch returns JSON representations of
fields with the "date" type. My confusion comes from the fact that the page
http://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-core-types.html
says:

"The date type is a special type which maps to JSON string type. It follows
a specific format that can be explicitly set. All dates are UTC.
Internally, a date maps to a number type long, with the added parsing stage
from string to long and from long to string.
" (emphasis mine)

It sounds like dates are stored as type 'long'. But when I POST documents
with dates and then retrieve them, they are returned in the same format as
I POSTed them. So it appears ES is storing how I POSTed each date
somewhere.

I have a reproducible curl example demonstrating my confusion in more
detail on Stackoverflow here:

Thank you for any insights!

Best,
--Erik Iverson

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/add6bafe-bdba-406e-b95d-ade17d0a9df5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Elastic won't edit your source. The long type is used internally

2015-03-20 14:16 GMT-03:00 Erik Iverson erikriverson@gmail.com:

Hello everyone,

I have a question on how Elasticsearch returns JSON representations of
fields with the "date" type. My confusion comes from the fact that the page
Field data types | Elasticsearch Guide [8.11] | Elastic
says:

"The date type is a special type which maps to JSON string type. It
follows a specific format that can be explicitly set. All dates are UTC.
Internally, a date maps to a number type long, with the added parsing stage
from string to long and from long to string.
" (emphasis mine)

It sounds like dates are stored as type 'long'. But when I POST documents
with dates and then retrieve them, they are returned in the same format as
I POSTed them. So it appears ES is storing how I POSTed each date
somewhere.

I have a reproducible curl example demonstrating my confusion in more
detail on Stackoverflow here:

datetime - How does Elasticsearch convert dates to JSON string representations? - Stack Overflow

Thank you for any insights!

Best,
--Erik Iverson

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/add6bafe-bdba-406e-b95d-ade17d0a9df5%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/add6bafe-bdba-406e-b95d-ade17d0a9df5%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJp2530Vwz99jV1K0JWNpGkOvR23PhHPAPacc-oROO0hZFjA0g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

It's Elasticsearch, Elastic is the company :slight_smile:

We convert dates to unix epoch, which is why you should insert them as UTC.

On 20 March 2015 at 10:22, Roger de Cordova Farias <
roger.farias@fontec.inf.br> wrote:

Elastic won't edit your source. The long type is used internally

2015-03-20 14:16 GMT-03:00 Erik Iverson erikriverson@gmail.com:

Hello everyone,

I have a question on how Elasticsearch returns JSON representations of
fields with the "date" type. My confusion comes from the fact that the page
Field data types | Elasticsearch Guide [8.11] | Elastic
says:

"The date type is a special type which maps to JSON string type. It
follows a specific format that can be explicitly set. All dates are
UTC. Internally, a date maps to a number type long, with the added parsing
stage from string to long and from long to string.
" (emphasis mine)

It sounds like dates are stored as type 'long'. But when I POST documents
with dates and then retrieve them, they are returned in the same format as
I POSTed them. So it appears ES is storing how I POSTed each date
somewhere.

I have a reproducible curl example demonstrating my confusion in more
detail on Stackoverflow here:

datetime - How does Elasticsearch convert dates to JSON string representations? - Stack Overflow

Thank you for any insights!

Best,
--Erik Iverson

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/add6bafe-bdba-406e-b95d-ade17d0a9df5%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/add6bafe-bdba-406e-b95d-ade17d0a9df5%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJp2530Vwz99jV1K0JWNpGkOvR23PhHPAPacc-oROO0hZFjA0g%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJp2530Vwz99jV1K0JWNpGkOvR23PhHPAPacc-oROO0hZFjA0g%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X84ecBBxOfBFNkKt3EKLSUiTCHAkqpv4fV44W30dYx-cw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Well, the company won't edit his source anyway :stuck_out_tongue: (but I get your point,
I'm used to refer to Elasticsearch as Elastic, I have to fix it)

I think his question is: he posts a document with a date in string format
and retrieve it in the same format. He was expecting to retrieve it as long
type since it is the type Elasticsearch uses internally.

I'm not familiar with the internal code of Elasticsearch, but as far as I
know, it won't change the source during indexing. It probably uses long
type in the index, but when you retrieve the source, you retrieve the
exactly source you posted

2015-03-20 16:16 GMT-03:00 Mark Walkom markwalkom@gmail.com:

It's Elasticsearch, Elastic is the company :slight_smile:

We convert dates to unix epoch, which is why you should insert them as UTC.

On 20 March 2015 at 10:22, Roger de Cordova Farias <
roger.farias@fontec.inf.br> wrote:

Elastic won't edit your source. The long type is used internally

2015-03-20 14:16 GMT-03:00 Erik Iverson erikriverson@gmail.com:

Hello everyone,

I have a question on how Elasticsearch returns JSON representations of
fields with the "date" type. My confusion comes from the fact that the page
Field data types | Elasticsearch Guide [8.11] | Elastic
says:

"The date type is a special type which maps to JSON string type. It
follows a specific format that can be explicitly set. All dates are
UTC. Internally, a date maps to a number type long, with the added parsing
stage from string to long and from long to string.
" (emphasis mine)

It sounds like dates are stored as type 'long'. But when I POST
documents with dates and then retrieve them, they are returned in the same
format as I POSTed them. So it appears ES is storing how I POSTed each date
somewhere.

I have a reproducible curl example demonstrating my confusion in more
detail on Stackoverflow here:

datetime - How does Elasticsearch convert dates to JSON string representations? - Stack Overflow

Thank you for any insights!

Best,
--Erik Iverson

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/add6bafe-bdba-406e-b95d-ade17d0a9df5%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/add6bafe-bdba-406e-b95d-ade17d0a9df5%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJp2530Vwz99jV1K0JWNpGkOvR23PhHPAPacc-oROO0hZFjA0g%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJp2530Vwz99jV1K0JWNpGkOvR23PhHPAPacc-oROO0hZFjA0g%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X84ecBBxOfBFNkKt3EKLSUiTCHAkqpv4fV44W30dYx-cw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X84ecBBxOfBFNkKt3EKLSUiTCHAkqpv4fV44W30dYx-cw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJp2533uGXxG_drPabJN48d_7ZgbXxhLKZaxnRrm%2BK3-QuvmaQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Ah ok, we convert between these formats so it's completely transparent to
the end user and easier to store under the hood.

On 20 March 2015 at 12:39, Roger de Cordova Farias <
roger.farias@fontec.inf.br> wrote:

Well, the company won't edit his source anyway :stuck_out_tongue: (but I get your point,
I'm used to refer to Elasticsearch as Elastic, I have to fix it)

I think his question is: he posts a document with a date in string format
and retrieve it in the same format. He was expecting to retrieve it as long
type since it is the type Elasticsearch uses internally.

I'm not familiar with the internal code of Elasticsearch, but as far as I
know, it won't change the source during indexing. It probably uses long
type in the index, but when you retrieve the source, you retrieve the
exactly source you posted

2015-03-20 16:16 GMT-03:00 Mark Walkom markwalkom@gmail.com:

It's Elasticsearch, Elastic is the company :slight_smile:

We convert dates to unix epoch, which is why you should insert them as
UTC.

On 20 March 2015 at 10:22, Roger de Cordova Farias <
roger.farias@fontec.inf.br> wrote:

Elastic won't edit your source. The long type is used internally

2015-03-20 14:16 GMT-03:00 Erik Iverson erikriverson@gmail.com:

Hello everyone,

I have a question on how Elasticsearch returns JSON representations of
fields with the "date" type. My confusion comes from the fact that the page
Field data types | Elasticsearch Guide [8.11] | Elastic
says:

"The date type is a special type which maps to JSON string type. It
follows a specific format that can be explicitly set. All dates are
UTC. Internally, a date maps to a number type long, with the added parsing
stage from string to long and from long to string.
" (emphasis mine)

It sounds like dates are stored as type 'long'. But when I POST
documents with dates and then retrieve them, they are returned in the same
format as I POSTed them. So it appears ES is storing how I POSTed each date
somewhere.

I have a reproducible curl example demonstrating my confusion in more
detail on Stackoverflow here:

datetime - How does Elasticsearch convert dates to JSON string representations? - Stack Overflow

Thank you for any insights!

Best,
--Erik Iverson

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/add6bafe-bdba-406e-b95d-ade17d0a9df5%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/add6bafe-bdba-406e-b95d-ade17d0a9df5%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJp2530Vwz99jV1K0JWNpGkOvR23PhHPAPacc-oROO0hZFjA0g%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJp2530Vwz99jV1K0JWNpGkOvR23PhHPAPacc-oROO0hZFjA0g%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X84ecBBxOfBFNkKt3EKLSUiTCHAkqpv4fV44W30dYx-cw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X84ecBBxOfBFNkKt3EKLSUiTCHAkqpv4fV44W30dYx-cw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJp2533uGXxG_drPabJN48d_7ZgbXxhLKZaxnRrm%2BK3-QuvmaQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAJp2533uGXxG_drPabJN48d_7ZgbXxhLKZaxnRrm%2BK3-QuvmaQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8hSoWzh2W0GLH8k0Tbbn_SZtqOseV2wO9DMARx6b57wQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.