# Reading gateway data in HDFS independently of ES **URL:** \ **Category:** Elasticsearch\ **Created:** [March 6, 2012, 4:55pm UTC](https://discuss.elastic.co/t/reading-gateway-data-in-hdfs-independently-of-es/6916 "2012-03-06T16:55:05Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Andrew\_Clegg](https://avatars.discourse-cdn.com/v4/letter/a/a88e4f/32.png) [@Andrew\_Clegg](https://discuss.elastic.co/u/Andrew_Clegg)\ **Post date:** [March 6, 2012, 4:55pm UTC](https://discuss.elastic.co/t/reading-gateway-data-in-hdfs-independently-of-es/6916/1 "2012-03-06T16:55:05Z")
Hi, If you use the Hadoop gateway to ship all your ES data to HDFS, is it in a format amenable to running map-reduce jobs over, independently of ES? For example, it would be really useful to be able to do Pig queries over the raw JSON document contents. Wonderdog ([https://github.com/](https://github.com/) infochimps/wonderdog) lets you do this via the ES cluster as a scan query, but that will put load on ES. If the data's already being written to the Hadoop cluster via a gateway, can you just analyse it there? And if so, does anyone have an example? Many thanks, Andrew. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 6, 2012, 8:56pm UTC](https://discuss.elastic.co/t/reading-gateway-data-in-hdfs-independently-of-es/6916/2 "2012-03-06T20:56:19Z")
No, its not really storing it in a way that you can easily read the actual json document. On Tuesday, March 6, 2012 at 6:55 PM, Andrew Clegg wrote: > Hi, > > If you use the Hadoop gateway to ship all your ES data to HDFS, is it > in a format amenable to running map-reduce jobs over, independently of > ES? > > For example, it would be really useful to be able to do Pig queries > over the raw JSON document contents. Wonderdog ([https://github.com/](https://github.com/) > infochimps/wonderdog) lets you do this via the ES cluster as a scan > query, but that will put load on ES. If the data's already being > written to the Hadoop cluster via a gateway, can you just analyse it > there? And if so, does anyone have an example? > > Many thanks, > > Andrew. ---
**Author:** ![Craig\_Brown](https://avatars.discourse-cdn.com/v4/letter/c/ce7236/32.png) [@Craig\_Brown](https://discuss.elastic.co/u/Craig_Brown)\ **Post date:** [March 6, 2012, 9:27pm UTC](https://discuss.elastic.co/t/reading-gateway-data-in-hdfs-independently-of-es/6916/3 "2012-03-06T21:27:32Z")
Could you read the indices with the lucene libraries? - Craig On Tue, Mar 6, 2012 at 1:56 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > No, its not really storing it in a way that you can easily read the > actual json document. > > On Tuesday, March 6, 2012 at 6:55 PM, Andrew Clegg wrote: > > Hi, > > If you use the Hadoop gateway to ship all your ES data to HDFS, is it > in a format amenable to running map-reduce jobs over, independently of > ES? > > For example, it would be really useful to be able to do Pig queries > over the raw JSON document contents. Wonderdog ([https://github.com/](https://github.com/) > infochimps/wonderdog) lets you do this via the ES cluster as a scan > query, but that will put load on ES. If the data's already being > written to the Hadoop cluster via a gateway, can you just analyse it > there? And if so, does anyone have an example? > > Many thanks, > > Andrew. -- … CRAIG BROWN chief architect youwho, Inc. _[www.youwho.com](http://www.youwho.com)_ [http://www.youwho.com/](http://www.youwho.com/) T: 801.855. 0921 M: 801.913. 0939 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:37am UTC](https://discuss.elastic.co/t/reading-gateway-data-in-hdfs-independently-of-es/6916/4 "2017-07-06T03:37:01Z")