# Elasticsearch-Hadoop: EsOutputFormat and the 'date' type

**URL:** <https://discuss.elastic.co/t/elasticsearch-hadoop-esoutputformat-and-the-date-type/18425>\
**Category:** Elasticsearch\
**Created:** [July 1, 2014, 8:09pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-esoutputformat-and-the-date-type/18425 "2014-07-01T20:09:57Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Telax](https://avatars.discourse-cdn.com/v4/letter/t/d26b3c/32.png) [@Telax](https://discuss.elastic.co/u/Telax)\
**Post date:** [July 1, 2014, 8:09pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-esoutputformat-and-the-date-type/18425/1 "2014-07-01T20:09:57Z")

</div>

Hello,

I'm interested in using the EsOutputFormat class in a hadoop mapreduce  
task.  
During experimentation I have noticed that there is no direct handling for  
'date' objects.  
My data contains a number of 'date' fields which must be transposed into  
the Elasticsearch index, however, I am currently unable to successfully  
transpose those fields which should be of type 'date' as instead they are  
simple submitted into the index as 'string' type.  
Using templates, I have tried to define a dynamic\_date\_formats as well as  
explicitly specifying a date type and format mapping for a matched field in  
a dynamic template which matches against the name of those fields which  
should be 'date' types.  
In either case, data fields indexed into my Elasticsearch cluster which  
should be recognized as 'date' types are only set as strings .

Here is an example template similar to that with which I have been  
experimenting.

{  
"template" : "index-name-_",  
"mappings" : {  
"default" : {  
"dynamic\_date\_formats" : ["yyyy-MM-dd hh:mm"]  
"dynamic\_templates" : [  
{ "date\_field\_template": {  
"match": "date\__",  
"mapping": {  
"type": "date",  
"format" : ""yyyy-MM-dd hh:mm""  
}  
}  
}  
}

Any help on this issue would be greatly appreciated.  
Thanks

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/10ff4422-ccdb-4fc0-8ccb-34b4b5e5180a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/10ff4422-ccdb-4fc0-8ccb-34b4b5e5180a%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\
**Post date:** [July 10, 2014, 9:52pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-esoutputformat-and-the-date-type/18425/2 "2014-07-10T21:52:05Z")

</div>

Make sure the template does match. This might not be always obvious however it's easy to test out. First, check your  
template and after defining the template, send a request with a sample payload to see whether the doc gets properly  
created. A common mistake is defining the template after the index is created which makes it useless; the template gets  
applied when a the index is created (and thus it becomes part of its mapping).  
Second, if the mapping appears correct, double-check your es-hadoop configuration and potentially turn on logging to see  
the payload sent by es-hadoop to elasticsearch.

Hope this helps,

On 7/1/14 11:09 PM, Telax wrote:

> Hello,
> 
> I'm interested in using the EsOutputFormat class in a hadoop mapreduce task.  
> During experimentation I have noticed that there is no direct handling for 'date' objects.  
> My data contains a number of 'date' fields which must be transposed into the Elasticsearch index, however, I am  
> currently unable to successfully transpose those fields which should be of type 'date' as instead they are simple  
> submitted into the index as 'string' type.  
> Using templates, I have tried to define a dynamic\_date\_formats as well as explicitly specifying a date type and format  
> mapping for a matched field in a dynamic template which matches against the name of those fields which should be 'date'  
> types.  
> In either case, data fields indexed into my Elasticsearch cluster which should be recognized as 'date' types are only  
> set as strings .
> 
> Here is an example template similar to that with which I have been experimenting.  
> {  
> "template" : "index-name-_",  
> "mappings" : {  
> "default" : {  
> "dynamic\_date\_formats" : ["yyyy-MM-dd hh:mm"]  
> "dynamic\_templates" : [  
> { "date\_field\_template": {  
> "match": "date\__",  
> "mapping": {  
> "type": "date",  
> "format" : ""yyyy-MM-dd hh:mm""  
> }  
> }  
> }  
> }
> 
> Any help on this issue would be greatly appreciated.  
> Thanks
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/10ff4422-ccdb-4fc0-8ccb-34b4b5e5180a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/10ff4422-ccdb-4fc0-8ccb-34b4b5e5180a%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/10ff4422-ccdb-4fc0-8ccb-34b4b5e5180a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/10ff4422-ccdb-4fc0-8ccb-34b4b5e5180a%40googlegroups.com?utm_medium=email&utm_source=footer).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
Costin

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/53BF0B05.6060205%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/53BF0B05.6060205%40gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Telax](https://avatars.discourse-cdn.com/v4/letter/t/d26b3c/32.png) [@Telax](https://discuss.elastic.co/u/Telax)\
**Post date:** [July 12, 2014, 6:08am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-esoutputformat-and-the-date-type/18425/3 "2014-07-12T06:08:03Z")

</div>

Hi Costin, thank you for your reply.

My issue actually came down to the ordering of my matches. I had a  
'match:\*' as the first dynamic template which disabled norms. Although this  
template didn't explicitly define a type for any matched field it would  
automatically set the 'date' field to a string type. The "date\_" template  
would then match but fail to set the type to date as it had already been  
defined. Simply reordering my dynamic templates so that the date matcher  
came before the catch all solved the issue 🙂  
On 10 Jul 2014 22:52, "Costin Leau" [costin.leau@gmail.com](mailto:costin.leau@gmail.com) wrote:

> Make sure the template does match. This might not be always obvious  
> however it's easy to test out. First, check your template and after  
> defining the template, send a request with a sample payload to see whether  
> the doc gets properly created. A common mistake is defining the template  
> after the index is created which makes it useless; the template gets  
> applied when a the index is created (and thus it becomes part of its  
> mapping).  
> Second, if the mapping appears correct, double-check your es-hadoop  
> configuration and potentially turn on logging to see the payload sent by  
> es-hadoop to elasticsearch.
> 
> Hope this helps,
> 
> On 7/1/14 11:09 PM, Telax wrote:
> 
> > Hello,
> > 
> > I'm interested in using the EsOutputFormat class in a hadoop mapreduce  
> > task.  
> > During experimentation I have noticed that there is no direct handling  
> > for 'date' objects.  
> > My data contains a number of 'date' fields which must be transposed into  
> > the Elasticsearch index, however, I am  
> > currently unable to successfully transpose those fields which should be  
> > of type 'date' as instead they are simple  
> > submitted into the index as 'string' type.  
> > Using templates, I have tried to define a dynamic\_date\_formats as well as  
> > explicitly specifying a date type and format  
> > mapping for a matched field in a dynamic template which matches against  
> > the name of those fields which should be 'date'  
> > types.  
> > In either case, data fields indexed into my Elasticsearch cluster which  
> > should be recognized as 'date' types are only  
> > set as strings .
> > 
> > Here is an example template similar to that with which I have been  
> > experimenting.  
> > {  
> > "template" : "index-name-_",  
> > "mappings" : {  
> > "default" : {  
> > "dynamic\_date\_formats" : ["yyyy-MM-dd hh:mm"]  
> > "dynamic\_templates" : [  
> > { "date\_field\_template": {  
> > "match": "date\__",  
> > "mapping": {  
> > "type": "date",  
> > "format" : ""yyyy-MM-dd hh:mm""  
> > }  
> > }  
> > }  
> > }
> > 
> > Any help on this issue would be greatly appreciated.  
> > Thanks
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to  
> > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \<mailto:elasticsearch+  
> > [unsubscribe@googlegroups.com](mailto:unsubscribe@googlegroups.com)\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/10ff4422-](https://groups.google.com/d/msgid/elasticsearch/10ff4422-)  
> > ccdb-4fc0-8ccb-34b4b5e5180a%[40googlegroups.com](http://40googlegroups.com)  
> > \<[https://groups.google.com/d/msgid/elasticsearch/10ff4422-](https://groups.google.com/d/msgid/elasticsearch/10ff4422-)  
> > ccdb-4fc0-8ccb-34b4b5e5180a%[40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players](http://40googlegroups.com?utm_medium=)  
> > email&utm\_source=footer\>.  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> Costin
> 
> --  
> You received this message because you are subscribed to a topic in the  
> Google Groups "elasticsearch" group.  
> To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> topic/elasticsearch/WPT086\_Q1ZI/unsubscribe.  
> To unsubscribe from this group and all its topics, send an email to  
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> msgid/elasticsearch/53BF0B05.6060205%[40gmail.com](http://40gmail.com).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAO4iR%2BHyAtqNF%2B0mGsz6-W9JkED7E6mhaeW4wd1\_v%2BpP6%3DUA-w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAO4iR%2BHyAtqNF%2B0mGsz6-W9JkED7E6mhaeW4wd1_v%2BpP6%3DUA-w%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:16am UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-esoutputformat-and-the-date-type/18425/4 "2017-07-06T01:16:19Z")

</div>