# How to index data when Hadoop and ES are in different clusters **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [May 26, 2015, 5:10pm UTC](https://discuss.elastic.co/t/how-to-index-data-when-hadoop-and-es-are-in-different-clusters/1332 "2015-05-26T17:10:59Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Kai\_Liu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kai_liu/32/404_2.png) [@Kai\_Liu](https://discuss.elastic.co/u/Kai_Liu)\ **Post date:** [May 26, 2015, 5:10pm UTC](https://discuss.elastic.co/t/how-to-index-data-when-hadoop-and-es-are-in-different-clusters/1332/1 "2015-05-26T17:10:59Z")
We want to make our data (~10TB) on HDFS interactive query-able, due to the data policy, we won't be able to install ES in the same cluster of Hadoop, so es-hadoop connector is not an option for us. Currently I'm trying to generate index data on Hadoop cluster, cause we have a relatively larger Hadoop cluster (~3K) comparing to ES cluster (up to 20). But have no idea about the size of index data and how to make it available on ES cluster yet (pull via HDFS proxy?). Do you have any idea? Thanks, Kai ---
**Author:** ![elastic\_paul](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@elastic\_paul](https://discuss.elastic.co/u/elastic_paul)\ **Post date:** [May 27, 2015, 7:09am UTC](https://discuss.elastic.co/t/how-to-index-data-when-hadoop-and-es-are-in-different-clusters/1332/2 "2015-05-27T07:09:03Z")
Hi Kai, I am not qualified to reply, but I will share what I am doing just for your interest. My hadoop cluster is 12 big nodes. Each node has 24 cores and 36TB. My ES cluster is 4 big nodes. Same as above All interconnected with 10G. On the 12 hadoop nodes I run Logstash. These point to the ES nodes. 3 hadoop nodes per ES node. eg; hadoop1 ---\> ES1 hadoop2 ---\> ES1 hadoop3 ---\> ES1 hadoop4 ---\> ES2 hadoop5 ---\> ES2 On the hadoop side I run a simple streaming mapper that basically does this: cat | nc localhost 3333 Note the use of netcat to direct the data to the local logstash on port 3333. Logstash then does everything and sends the data using the Bulk API. I have different log sources and it just needs a change of port number for the netcat command. Logstash listens on the various ports, 3333, 3334, 3335 etc Not very sophisticated but it works, and it pretty easy. ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [June 4, 2015, 6:50pm UTC](https://discuss.elastic.co/t/how-to-index-data-when-hadoop-and-es-are-in-different-clusters/1332/3 "2015-06-04T18:50:16Z")
The setup form `elastic_paul` works. Additionally es-hadoop connector does _not_ require Elasticsearch to sit next to Hadoop. In can be anywhere where the Hadoop nodes can have access to it - note that the connector supports HTTPS and SOCKS proxies so even in restricted environments, one can route traffic through a certain direction. A nice thing about Elasticsearch (I'm biased) is that you can play fairly easy with its topology - you can install nodes on the same machine or spread them out. You can even run multiple Elasticsearch clusters on the same machines, no problem. With Hadoop, one doesn't lose that - you can run it entirely on the same machines as Hadoop, just some nodes or no nodes at all. And the connector doesn't mind - in fact, it supports all the cases above. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:28pm UTC](https://discuss.elastic.co/t/how-to-index-data-when-hadoop-and-es-are-in-different-clusters/1332/4 "2017-07-06T13:28:19Z")