# HDFS storage options

**URL:** https://discuss.elastic.co/t/hdfs-storage-options/38418
**Category:** Elasticsearch
**Tags:** es-hadoop
**Created:** [January 5, 2016, 3:21pm UTC](https://discuss.elastic.co/t/hdfs-storage-options/38418 "2016-01-05T15:21:29Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![Johannes\_Zillmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johannes_zillmann/32/6939_2.png) [@Johannes\_Zillmann](https://discuss.elastic.co/u/Johannes_Zillmann)
#### Post date: [January 5, 2016, 3:21pm UTC](https://discuss.elastic.co/t/hdfs-storage-options/38418/1 "2016-01-05T15:21:29Z")

</div>

I spend a couple of time to find out how ES can possibly integrate to HDFS.  
We have an ES cluster running on top of YARN and want the cluster to be fail safe, e.g. survive a YARN restart.

My conclusion is:

- (1) you can mount HDFS as NFS and point ES to a NFS path (downside: slowdown)
- (2) you can use [repository-hdfs](https://github.com/elastic/elasticsearch-hadoop/tree/master/repository-hdfs) and 'manually' care about backup and restore to and from HDFS

Any other options ?  
Also i'm yet un-decided on whether to use ES 1.x or 2.x, does it matter in that perspective ?  
best  
Johannes

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 5, 2016, 8:46pm UTC](https://discuss.elastic.co/t/hdfs-storage-options/38418/2 "2016-01-05T20:46:08Z")

</div>

> [@Johannes\_Zillmann](#):
>
> - (1) you can mount HDFS as NFS and point ES to a NFS path (downside: slowdown)

This will be _really_ slow, to the point where it'd be unusable, and we do not recommend it.

---

<div class="post-metadata">

### Author: ![Johannes\_Zillmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johannes_zillmann/32/6939_2.png) [@Johannes\_Zillmann](https://discuss.elastic.co/u/Johannes_Zillmann)
#### Post date: [January 7, 2016, 10:08am UTC](https://discuss.elastic.co/t/hdfs-storage-options/38418/3 "2016-01-07T10:08:49Z")

</div>

Ok, understood. Don't use option (1) / NFS-HDFS.  
So but thats all my options ? There isn't an option (3) where all my data is persisted in HDFS but the nodes operate on a local copy or anything like that !?

---

<div class="post-metadata">

### Author: ![ssatapathy](https://avatars.discourse-cdn.com/v4/letter/s/58f4c7/32.png) [@ssatapathy](https://discuss.elastic.co/u/ssatapathy)
#### Post date: [January 7, 2016, 2:20pm UTC](https://discuss.elastic.co/t/hdfs-storage-options/38418/4 "2016-01-07T14:20:20Z")

</div>

Actually you can. One can have HDFS as the primary storage and upload the data from HDSF to ES, where data on ES can exist in local nodes.

---

<div class="post-metadata">

### Author: ![Johannes\_Zillmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johannes_zillmann/32/6939_2.png) [@Johannes\_Zillmann](https://discuss.elastic.co/u/Johannes_Zillmann)
#### Post date: [January 14, 2016, 9:41am UTC](https://discuss.elastic.co/t/hdfs-storage-options/38418/5 "2016-01-14T09:41:28Z")

</div>

And how is that configured ?

---

<div class="post-metadata">

### Author: ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)
#### Post date: [January 14, 2016, 3:37pm UTC](https://discuss.elastic.co/t/hdfs-storage-options/38418/6 "2016-01-14T15:37:03Z")

</div>

What I think @ssatapathy is suggesting is to keep your data in HDFS (primary storage) and load it through Hadoop jobs into ES. ES is using the out of the box configuration, writing data to the local disks/storage.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:26pm UTC](https://discuss.elastic.co/t/hdfs-storage-options/38418/7 "2017-07-06T13:26:30Z")

</div>


