Joda date format error

Elastic Stack Elasticsearch
1

Hi,

While applying mappings for a date field, I tried the following in the
date_formats field.
"MM/yyyy", "yyyy/MM/dd HH:mm:ss"

When I submitted content for indexing, this format was generating an error.

java.lang.IllegalArgumentException: Invalid format: "05/2010" is
malformed at "/2010"
at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:634)

In the second case, it was "2010/05/03 10:43:00" is malformed at "/05/03"

I noticed that this error is coming up for formats where a slash (/)
is used like yyyy/MM/dd or MM/yyyy

From the Joda documentation and some examples, I figured that formats
like this are valid.

Could this possibly be a bug somewhere or am I missing something ?

Regards,
Mahendra

http://twitter.com/mahendra

2

Hi,

Can you post the mapping you use?

cheers,
shay.banon

On May 3, 4:40 pm, Mahendra M mahendra.m@gmail.com wrote:

Hi,

While applying mappings for a date field, I tried the following in the
date_formats field.
"MM/yyyy", "yyyy/MM/dd HH:mm:ss"

When I submitted content for indexing, this format was generating an error.

java.lang.IllegalArgumentException: Invalid format: "05/2010" is
malformed at "/2010"
at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:6 34)

In the second case, it was "2010/05/03 10:43:00" is malformed at "/05/03"

I noticed that this error is coming up for formats where a slash (/)
is used like yyyy/MM/dd or MM/yyyy

From the Joda documentation and some examples, I figured that formats
like this are valid.

Could this possibly be a bug somewhere or am I missing something ?

Regards,
Mahendra

http://twitter.com/mahendra

3

On Mon, May 3, 2010 at 7:39 PM, Shay Banon shay.banon@elasticsearch.com wrote:

Hi,

Can you post the mapping you use?

Hi,

The JSON is as follows - this is a test one. The field 'genre' is
marked as date.

{
"a" : {
"type" : "object",
"dynamic" : false,
"enabled" : true,
"path" : "full",
"date_formats" : [ "yyyy/MM/dd" ],
"_boost" : {
"name" : "_boost"
},
"_id" : {
"store" : "no"
},
"_all" : {
"enabled" : true,
"store" : "no",
"term_vector" : "no"
},
"_source" : {
"name" : "_source",
"enabled" : true
},
"properties" : {
"genre" : {
"type" : "date",
"date_formats" : [ "yyyy/MM/dd" ],
"index_name" : "genre",
"index" : "analyzed",
"store" : "no",
"term_vector" : "no",
"boost" : 1.0,
"omit_norms" : false,
"include_in_all": false,
"omit_term_freq_and_positions" : false
},
"sub_genre" : {
"type" : "string",
"index_name" : "sub_genre",
"index" : "analyzed",
"store" : "no",
"term_vector" : "no",
"boost" : 1.0,
"omit_norms" : false,
"include_in_all": false,
"omit_term_freq_and_positions" : false
}
}
}
}

The following content fails here:
{
"genre": "2010/05/03",
"sub_genre": "test"
}

PS: I am trying this on elasticsearch 0.6

Regards,
Mahendra

http://twitter.com/mahendra

4

Hi,

How I made the JSON was like this.

  • Index a document without any mappings.
  • Check elasticsearch logs for the mappings that were generated dynamically.
  • Modify the mappings according to my needs.

I hope this is the correct approach..

Regards,
Mahendra

On Mon, May 3, 2010 at 9:57 PM, Mahendra M mahendra.m@gmail.com wrote:

Hi,

The JSON is as follows - this is a test one. The field 'genre' is
marked as date.

{
"a" : {
"type" : "object",
"dynamic" : false,
"enabled" : true,
"path" : "full",
"date_formats" : [ "yyyy/MM/dd" ],
"_boost" : {
"name" : "_boost"
},
"_id" : {
"store" : "no"
},
"_all" : {
"enabled" : true,
"store" : "no",
"term_vector" : "no"
},
"_source" : {
"name" : "_source",
"enabled" : true
},
"properties" : {
"genre" : {
"type" : "date",
"date_formats" : [ "yyyy/MM/dd" ],
"index_name" : "genre",
"index" : "analyzed",
"store" : "no",
"term_vector" : "no",
"boost" : 1.0,
"omit_norms" : false,
"include_in_all": false,
"omit_term_freq_and_positions" : false
},
"sub_genre" : {
"type" : "string",
"index_name" : "sub_genre",
"index" : "analyzed",
"store" : "no",
"term_vector" : "no",
"boost" : 1.0,
"omit_norms" : false,
"include_in_all": false,
"omit_term_freq_and_positions" : false
}
}
}
}

The following content fails here:
{
"genre": "2010/05/03",
"sub_genre": "test"
}

PS: I am trying this on elasticsearch 0.6

Regards,
Mahendra

http://twitter.com/mahendra

5

Hi,

The mapping for specifying a format on a date field is called 'format' not
'data_formats'. The 'date_formats' only applies to the object mapping, and
is relevant when a new field is introduced that has no mappings in order to
try and derive if its a date type or not.

Regarding the automatic detection of date fields based on the object
mapping, the format you specified in the date_formats section still won't
make date types to be detected properly. This is since I also check for the
existence of '-' or ':' in the string. This is because the date parser is
too lenient and manages to parse things that should be string with just
numbers in them ;). I will add the '/' to it as well while I think of a
better way to work around this...

cheers,
shay.banon

On Tue, May 4, 2010 at 11:19 AM, Mahendra M mahendra.m@gmail.com wrote:

Hi,

How I made the JSON was like this.

  • Index a document without any mappings.
  • Check elasticsearch logs for the mappings that were generated
    dynamically.
  • Modify the mappings according to my needs.

I hope this is the correct approach..

Regards,
Mahendra

On Mon, May 3, 2010 at 9:57 PM, Mahendra M mahendra.m@gmail.com wrote:

Hi,

The JSON is as follows - this is a test one. The field 'genre' is
marked as date.

{
"a" : {
"type" : "object",
"dynamic" : false,
"enabled" : true,
"path" : "full",
"date_formats" : [ "yyyy/MM/dd" ],
"_boost" : {
"name" : "_boost"
},
"_id" : {
"store" : "no"
},
"_all" : {
"enabled" : true,
"store" : "no",
"term_vector" : "no"
},
"_source" : {
"name" : "_source",
"enabled" : true
},
"properties" : {
"genre" : {
"type" : "date",
"date_formats" : [ "yyyy/MM/dd" ],
"index_name" : "genre",
"index" : "analyzed",
"store" : "no",
"term_vector" : "no",
"boost" : 1.0,
"omit_norms" : false,
"include_in_all": false,
"omit_term_freq_and_positions" : false
},
"sub_genre" : {
"type" : "string",
"index_name" : "sub_genre",
"index" : "analyzed",
"store" : "no",
"term_vector" : "no",
"boost" : 1.0,
"omit_norms" : false,
"include_in_all": false,
"omit_term_freq_and_positions" : false
}
}
}
}

The following content fails here:
{
"genre": "2010/05/03",
"sub_genre": "test"
}

PS: I am trying this on elasticsearch 0.6

Regards,
Mahendra

http://twitter.com/mahendra

6

On Tue, May 4, 2010 at 5:56 PM, Shay Banon shay.banon@elasticsearch.com wrote:

I will add the '/' to it as well while I think of a
better way to work around this...

That would be cool. I am actually writing a couchdb module to
integrate it with elasticsearch (will apply to my company for
opensourcing it). All of couchdb examples and documents use the date
fields with a '/' (saying parsing it is easy in JS)

Format is : yyyy/MM/dd HH:mm:ss +SSSS

Having support for this will make couchdb integration a lot smoother.

Regards,
Mahendra

http://twitter.com/mahendra

7

Already in master (the support for auto detection of date fields with '/'),
but, still the default date_formats for objects is the ISO one, we can make
the integration even smoother if I add this format to the list of default
date_formats if none are specified... .

On Wed, May 5, 2010 at 9:16 AM, Mahendra M mahendra.m@gmail.com wrote:

On Tue, May 4, 2010 at 5:56 PM, Shay Banon shay.banon@elasticsearch.com
wrote:

I will add the '/' to it as well while I think of a
better way to work around this...

That would be cool. I am actually writing a couchdb module to
integrate it with elasticsearch (will apply to my company for
opensourcing it). All of couchdb examples and documents use the date
fields with a '/' (saying parsing it is easy in JS)

Format is : yyyy/MM/dd HH:mm:ss +SSSS

Having support for this will make couchdb integration a lot smoother.

Regards,
Mahendra

http://twitter.com/mahendra

8

By the way, where in couchdb does is this format specified, I can't find it.
It seems like it all depends on how you input the string that represents the
date...

On Wed, May 5, 2010 at 11:27 AM, Shay Banon shay.banon@elasticsearch.comwrote:

Already in master (the support for auto detection of date fields with '/'),
but, still the default date_formats for objects is the ISO one, we can make
the integration even smoother if I add this format to the list of default
date_formats if none are specified... .

On Wed, May 5, 2010 at 9:16 AM, Mahendra M mahendra.m@gmail.com wrote:

On Tue, May 4, 2010 at 5:56 PM, Shay Banon shay.banon@elasticsearch.com
wrote:

I will add the '/' to it as well while I think of a
better way to work around this...

That would be cool. I am actually writing a couchdb module to
integrate it with elasticsearch (will apply to my company for
opensourcing it). All of couchdb examples and documents use the date
fields with a '/' (saying parsing it is easy in JS)

Format is : yyyy/MM/dd HH:mm:ss +SSSS

Having support for this will make couchdb integration a lot smoother.

Regards,
Mahendra

http://twitter.com/mahendra

9

On Wed, May 5, 2010 at 2:11 PM, Shay Banon shay.banon@elasticsearch.com wrote:

By the way, where in couchdb does is this format specified, I can't find it.
It seems like it all depends on how you input the string that represents the
date...

It is not mandatory. The examples are mostly in "/" format. In
couchdb, the sorting happens based on string comparison. So, it does
not matter if it is /,- or :

You can see some examples here:
http://books.couchdb.org/relax/design-documents/views

--
Mahendra

http://twitter.com/mahendra

10

Committed: Mapper: Object mapper default 'date_formats' to also support 'yyyy/MM/dd HH:mm:ss' and 'yyyy/MM/dd' · Issue #159 · elastic/elasticsearch · GitHub.

On Wed, May 5, 2010 at 1:19 PM, Mahendra M mahendra.m@gmail.com wrote:

On Wed, May 5, 2010 at 2:11 PM, Shay Banon shay.banon@elasticsearch.com
wrote:

By the way, where in couchdb does is this format specified, I can't find
it.
It seems like it all depends on how you input the string that represents
the
date...

It is not mandatory. The examples are mostly in "/" format. In
couchdb, the sorting happens based on string comparison. So, it does
not matter if it is /,- or :

You can see some examples here:
http://books.couchdb.org/relax/design-documents/views

--
Mahendra

http://twitter.com/mahendra