# Building with Hadoop - best setup? **URL:** \ **Category:** Elasticsearch\ **Created:** [March 26, 2010, 2:35pm UTC](https://discuss.elastic.co/t/building-with-hadoop-best-setup/2871 "2010-03-26T14:35:53Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![timrobertson100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timrobertson100/32/2467_2.png) [@timrobertson100](https://discuss.elastic.co/u/timrobertson100)\ **Post date:** [March 26, 2010, 2:35pm UTC](https://discuss.elastic.co/t/building-with-hadoop-best-setup/2871/1 "2010-03-26T14:35:53Z")
Hey, Is anyone building their indexes using Hadoop? If so, are they deploying ES across the same cluster as Hadoop and trying to reduce network noise by making use of data locality, or keeping the clusters separate and just calling over HTTP from MapReduce when building the indexes? I am about to set up on EC2, and planned to keep the search and processing machines separate. Cheers, Tim ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 26, 2010, 2:58pm UTC](https://discuss.elastic.co/t/building-with-hadoop-best-setup/2871/2 "2010-03-26T14:58:26Z")
Hi, You won't enjoy locally between elasticsearch and hadoop in any case since both use different distribution model. The locality would only make sense for the indexing part, and think that you probably won't really need it (it should be fast enough). What language are you going to write your jobs at? If Java, then make use of the native Java client (obtained from a "non data" Server started) and not HTTP. More here: [http://www.elasticsearch.com/docs/elasticsearch/java\_api/client/#Server\_Client](http://www.elasticsearch.com/docs/elasticsearch/java_api/client/#Server_Client) -shay.banon On Fri, Mar 26, 2010 at 5:35 PM, Tim Robertson [timrobertson100@gmail.com](mailto:timrobertson100@gmail.com)wrote: > Hey, > > Is anyone building their indexes using Hadoop? If so, are they deploying > ES across the same cluster as Hadoop and trying to reduce network noise by > making use of data locality, or keeping the clusters separate and just > calling over HTTP from MapReduce when building the indexes? I am about to > set up on EC2, and planned to keep the search and processing machines > separate. > > Cheers, > Tim ---
**Author:** ![timrobertson100](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timrobertson100/32/2467_2.png) [@timrobertson100](https://discuss.elastic.co/u/timrobertson100)\ **Post date:** [March 26, 2010, 5:10pm UTC](https://discuss.elastic.co/t/building-with-hadoop-best-setup/2871/3 "2010-03-26T17:10:53Z")
Thanks Shay, So... reading between the lines, does it then use protobufs (or other?) for RPC instead of JSON and serializing and deserializing? Cheers Tim On Fri, Mar 26, 2010 at 3:58 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Hi, > > You won't enjoy locally between elasticsearch and hadoop in any case > since both use different distribution model. The locality would only make > sense for the indexing part, and think that you probably won't really need > it (it should be fast enough). > > What language are you going to write your jobs at? If Java, then make use > of the native Java client (obtained from a "non data" Server started) and > not HTTP. More here: > [http://www.elasticsearch.com/docs/elasticsearch/java\_api/client/#Server\_Client](http://www.elasticsearch.com/docs/elasticsearch/java_api/client/#Server_Client) > > -shay.banon > > On Fri, Mar 26, 2010 at 5:35 PM, Tim Robertson [timrobertson100@gmail.com](mailto:timrobertson100@gmail.com)wrote: > > > Hey, > > > > Is anyone building their indexes using Hadoop? If so, are they deploying > > ES across the same cluster as Hadoop and trying to reduce network noise by > > making use of data locality, or keeping the clusters separate and just > > calling over HTTP from MapReduce when building the indexes? I am about to > > set up on EC2, and planned to keep the search and processing machines > > separate. > > > > Cheers, > > Tim ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 26, 2010, 5:59pm UTC](https://discuss.elastic.co/t/building-with-hadoop-best-setup/2871/4 "2010-03-26T17:59:19Z")
With the Java client, the "source" (which is the indexable document) is still json, but everything around it is a highly optimized stream serialization/deserialization. It is internal and does not use protobuff, but I expect it to be at least 10x faster than protobuf. -shay.banon On Fri, Mar 26, 2010 at 8:10 PM, Tim Robertson [timrobertson100@gmail.com](mailto:timrobertson100@gmail.com)wrote: > Thanks Shay, > > So... reading between the lines, does it then use protobufs (or other?) for > RPC instead of JSON and serializing and deserializing? > > Cheers > Tim > > On Fri, Mar 26, 2010 at 3:58 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > > Hi, > > > > You won't enjoy locally between elasticsearch and hadoop in any case > > since both use different distribution model. The locality would only make > > sense for the indexing part, and think that you probably won't really need > > it (it should be fast enough). > > > > What language are you going to write your jobs at? If Java, then make > > use of the native Java client (obtained from a "non data" Server started) > > and not HTTP. More here: > > [http://www.elasticsearch.com/docs/elasticsearch/java\_api/client/#Server\_Client](http://www.elasticsearch.com/docs/elasticsearch/java_api/client/#Server_Client) > > > > -shay.banon > > > > On Fri, Mar 26, 2010 at 5:35 PM, Tim Robertson \<[timrobertson100@gmail.com](mailto:timrobertson100@gmail.com) > > > > > wrote: > > > > > Hey, > > > > > > Is anyone building their indexes using Hadoop? If so, are they deploying > > > ES across the same cluster as Hadoop and trying to reduce network noise by > > > making use of data locality, or keeping the clusters separate and just > > > calling over HTTP from MapReduce when building the indexes? I am about to > > > set up on EC2, and planned to keep the search and processing machines > > > separate. > > > > > > Cheers, > > > Tim ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 26, 2010, 6:02pm UTC](https://discuss.elastic.co/t/building-with-hadoop-best-setup/2871/5 "2010-03-26T18:02:03Z")
Just one note here thought, what takes most of the time in this cases is the remote call itself, not the serialization. But when it comes to pure serialization, its highly optimized. -shay.banon On Fri, Mar 26, 2010 at 8:59 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > With the Java client, the "source" (which is the indexable document) is > still json, but everything around it is a highly optimized stream > serialization/deserialization. It is internal and does not use protobuff, > but I expect it to be at least 10x faster than protobuf. > > -shay.banon > > On Fri, Mar 26, 2010 at 8:10 PM, Tim Robertson [timrobertson100@gmail.com](mailto:timrobertson100@gmail.com)wrote: > > > Thanks Shay, > > > > So... reading between the lines, does it then use protobufs (or other?) > > for RPC instead of JSON and serializing and deserializing? > > > > Cheers > > Tim > > > > On Fri, Mar 26, 2010 at 3:58 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) > > > > > wrote: > > > > > Hi, > > > > > > You won't enjoy locally between elasticsearch and hadoop in any case > > > since both use different distribution model. The locality would only make > > > sense for the indexing part, and think that you probably won't really need > > > it (it should be fast enough). > > > > > > What language are you going to write your jobs at? If Java, then make > > > use of the native Java client (obtained from a "non data" Server started) > > > and not HTTP. More here: > > > [http://www.elasticsearch.com/docs/elasticsearch/java\_api/client/#Server\_Client](http://www.elasticsearch.com/docs/elasticsearch/java_api/client/#Server_Client) > > > > > > -shay.banon > > > > > > On Fri, Mar 26, 2010 at 5:35 PM, Tim Robertson \< > > > [timrobertson100@gmail.com](mailto:timrobertson100@gmail.com)\> wrote: > > > > > > > Hey, > > > > > > > > Is anyone building their indexes using Hadoop? If so, are they > > > > deploying ES across the same cluster as Hadoop and trying to reduce network > > > > noise by making use of data locality, or keeping the clusters separate and > > > > just calling over HTTP from MapReduce when building the indexes? I am about > > > > to set up on EC2, and planned to keep the search and processing machines > > > > separate. > > > > > > > > Cheers, > > > > Tim ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:25am UTC](https://discuss.elastic.co/t/building-with-hadoop-best-setup/2871/6 "2017-07-06T04:25:04Z")
Closed on July 6, 2017, 4:25am UTC