# Elasticsearch with hadoop and pig **URL:** \ **Category:** Elasticsearch\ **Created:** [March 4, 2014, 2:56pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop-and-pig/16147 "2014-03-04T14:56:25Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![srivibalu](https://avatars.discourse-cdn.com/v4/letter/s/a587f6/32.png) [@srivibalu](https://discuss.elastic.co/u/srivibalu)\ **Post date:** [March 4, 2014, 2:56pm UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop-and-pig/16147/1 "2014-03-04T14:56:25Z")
I am doing a poc project using elasticsearch-hadoop and using pig to create and read the index. Is there an elaborate documentation on how to use elasticsearch with pig and hadoop? I want to know if the index could be saved in HDFS permanently or is HDFS just a backup store only? Also, load from es index command is not returning the schema (index mapping) in pig, so i am unable to reference the fields using the name in subsequent steps. filters work fine but sorting doesn't work. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8453cfd0-ea5c-4614-bf6d-e3c838e5980f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8453cfd0-ea5c-4614-bf6d-e3c838e5980f%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Yann\_Barraud](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yann_barraud/32/1108_2.png) [@Yann\_Barraud](https://discuss.elastic.co/u/Yann_Barraud)\ **Post date:** [March 5, 2014, 9:44am UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop-and-pig/16147/2 "2014-03-05T09:44:39Z")
Hi, I poublished this one last week. [https://github.com/hortonworks/hadoop-tutorials/blob/master/Community/T07\_Elasticsearch\_Hadoop\_Integration.md](https://github.com/hortonworks/hadoop-tutorials/blob/master/Community/T07_Elasticsearch_Hadoop_Integration.md) Le mardi 4 mars 2014 15:56:25 UTC+1, [sriv...@gmail.com](mailto:sriv...@gmail.com) a écrit : > I am doing a poc project using elasticsearch-hadoop and using pig to > create and read the index. Is there an elaborate documentation on how to > use elasticsearch with pig and hadoop? > I want to know if the index could be saved in HDFS permanently or is HDFS > just a backup store only? > > Also, load from es index command is not returning the schema (index > mapping) in pig, so i am unable to reference the fields using the name in > subsequent steps. filters work fine but sorting doesn't work. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/aeb2a1b7-6c88-4324-9003-2d1ab65add82%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aeb2a1b7-6c88-4324-9003-2d1ab65add82%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [March 5, 2014, 10:44am UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop-and-pig/16147/3 "2014-03-05T10:44:50Z")
Have you looked at the reference documentation [1] ? There also an overview video with the main es-hadoop features [2]. Regarding HDFS, you can mount it as a NFS and expose it to ES which can use it directly. As for the mapping, you need to define it in Pig as it will not be inferred from the ES mapping. This is done on purpose since his is how all libraries in Hadoop work as the data isn't guaranteed to be normalized. Additionally you typically want just a section of the data so the Hadoop-side mapping indicates the view applied to ES. In the end, the driver is Hadoop hence its settings take precedence. [1] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/pig.html) [2] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/videos/search-and-analytics-with-hadoop-and-elasticsearch/) On 3/4/2014 4:56 PM, [srivibalu@gmail.com](mailto:srivibalu@gmail.com) wrote: > I am doing a poc project using elasticsearch-hadoop and using pig to create and read the index. Is there an elaborate > documentation on how to use elasticsearch with pig and hadoop? > I want to know if the index could be saved in HDFS permanently or is HDFS just a backup store only? > Also, load from es index command is not returning the schema (index mapping) in pig, so i am unable to reference the > fields using the name in subsequent steps. filters work fine but sorting doesn't work. > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/8453cfd0-ea5c-4614-bf6d-e3c838e5980f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8453cfd0-ea5c-4614-bf6d-e3c838e5980f%40googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/53170022.7070701%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/53170022.7070701%40gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:45am UTC](https://discuss.elastic.co/t/elasticsearch-with-hadoop-and-pig/16147/4 "2017-07-06T01:45:37Z")