# Can elasticsearch reads and stores data in HDFS by es-hadoop?

**URL:** https://discuss.elastic.co/t/can-elasticsearch-reads-and-stores-data-in-hdfs-by-es-hadoop/40713
**Category:** Elasticsearch
**Tags:** es-hadoop
**Created:** [February 2, 2016, 9:05am UTC](https://discuss.elastic.co/t/can-elasticsearch-reads-and-stores-data-in-hdfs-by-es-hadoop/40713 "2016-02-02T09:05:58Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![Cancellala](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cancellala/32/14010_2.png) [@Cancellala](https://discuss.elastic.co/u/Cancellala)
#### Post date: [February 2, 2016, 9:05am UTC](https://discuss.elastic.co/t/can-elasticsearch-reads-and-stores-data-in-hdfs-by-es-hadoop/40713/1 "2016-02-02T09:05:58Z")

</div>

Hi all,  
I run elastic on YARN refer to this page:[https://www.elastic.co/guide/en/elasticsearch/hadoop/master/ey-usage.html#yarn-provision-es](https://www.elastic.co/guide/en/elasticsearch/hadoop/master/ey-usage.html#yarn-provision-es)

I saw that the elasticsearch service runs on datanode randomly, and the data of elasticsearch are stored locally, not in HDFS. How could this be?

As i see, the home page of es-hadoop show me that es-hadoop should make elasticsearhc can read and store data on hdfs, is it right ?

Could anybody please show me a examples of successful installation? better with installation documentation . i need it so much.

Thank you!

---

<div class="post-metadata">

### Author: ![Cancellala](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cancellala/32/14010_2.png) [@Cancellala](https://discuss.elastic.co/u/Cancellala)
#### Post date: [February 18, 2016, 7:53am UTC](https://discuss.elastic.co/t/can-elasticsearch-reads-and-stores-data-in-hdfs-by-es-hadoop/40713/2 "2016-02-18T07:53:46Z")

</div>

nobody can answer me? please~

---

<div class="post-metadata">

### Author: ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)
#### Post date: [February 21, 2016, 10:49pm UTC](https://discuss.elastic.co/t/can-elasticsearch-reads-and-stores-data-in-hdfs-by-es-hadoop/40713/3 "2016-02-21T22:49:23Z")

</div>

If one _really_ wants to run ES directly on HDFS they can do so right now by mounting HDFS as a local NFS partition. However it will not only be slow but also there might be some data loss due to the various semantics - HDFS is not an actual file system.

As for the data, it can be indexed and query from Hadoop (and its various libraries) in a native, parallel way through the ES-Hadoop connector.

---

<div class="post-metadata">

### Author: ![Cancellala](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cancellala/32/14010_2.png) [@Cancellala](https://discuss.elastic.co/u/Cancellala)
#### Post date: [February 26, 2016, 2:40am UTC](https://discuss.elastic.co/t/can-elasticsearch-reads-and-stores-data-in-hdfs-by-es-hadoop/40713/4 "2016-02-26T02:40:53Z")

</div>

Hi Costin,  
Many thanks for your kindly tips.  
Still have questions that:  
1)how could you solve the problem that elasticsearch service runs on datanode randomly? it confuses me for a long time.  
2)could you please show me a examples of successful installation? better with installation documentation

Best wishes.

---

<div class="post-metadata">

### Author: ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)
#### Post date: [March 3, 2016, 9:39am UTC](https://discuss.elastic.co/t/can-elasticsearch-reads-and-stores-data-in-hdfs-by-es-hadoop/40713/5 "2016-03-03T09:39:18Z")

</div>

You're welcome.

1. by taking care of provisioning by yourself. Puppet, chef, basic ssh+scripts, whatever it's easier for you to deploy ES on the given machines. YARN doesn't provide any type of provisioning and makes no guarantees of where a process (short-lived) runs.

2. see 1. If you really want to use YARN, the docs already contain examples of how to start and stop it. If you need finer grained control , etc.., then I'm afraid ES-YARN does not provide them, at least in its current form.

---

<div class="post-metadata">

### Author: ![krish0608](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krish0608/32/6797_2.png) [@krish0608](https://discuss.elastic.co/u/krish0608)
#### Post date: [May 23, 2016, 6:28am UTC](https://discuss.elastic.co/t/can-elasticsearch-reads-and-stores-data-in-hdfs-by-es-hadoop/40713/6 "2016-05-23T06:28:40Z")

</div>

Hi ,

I am Facing same problem as yours. If you have done it please let me know how can i figure out this problem.  
I have hadoop cluster my elasticsearch service randomly runs on datanode as yours but whenever i put data on my es it start storing it in default elasticsearch datapath.  
Please help me out i am stucked in it for long time.But In case of single hadoop node and single es its works fine.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:24pm UTC](https://discuss.elastic.co/t/can-elasticsearch-reads-and-stores-data-in-hdfs-by-es-hadoop/40713/7 "2017-07-06T13:24:38Z")

</div>


