# Using repository-hdfs on (different) YARN cluster

**URL:** <https://discuss.elastic.co/t/using-repository-hdfs-on-different-yarn-cluster/41414>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [February 10, 2016, 5:57pm UTC](https://discuss.elastic.co/t/using-repository-hdfs-on-different-yarn-cluster/41414 "2016-02-10T17:57:58Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Johannes\_Zillmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johannes_zillmann/32/6939_2.png) [@Johannes\_Zillmann](https://discuss.elastic.co/u/Johannes_Zillmann)\
**Post date:** [February 10, 2016, 5:57pm UTC](https://discuss.elastic.co/t/using-repository-hdfs-on-different-yarn-cluster/41414/1 "2016-02-10T17:57:58Z")

</div>

Hey there, question!

Given

- i have an ES cluster running on top of YARN via elasticsearch-yarn
- the ES cluster has the plugin repository-hdfs installed

What is the best way to ensure the plugin's hadoop-libs are matching those on the cluster ?

Checkout [https://github.com/elastic/elasticsearch-hadoop/tree/master/repository-hdfs](https://github.com/elastic/elasticsearch-hadoop/tree/master/repository-hdfs) and build the plugin with the correct flavour/hadoop-version (like described in [Issues with using repository-hdfs plug in for snapshot/restore operation](https://discuss.elastic.co/t/issues-with-using-repository-hdfs-plug-in-for-snapshot-restore-operation/18513/2))?

Or is there anyway to tell the plugin that it makes use of the YARN/Hadoop-Classpath instead of its own plugin-lib folder ? (that would be preferred because i want to run ES on various different Hadoop distributions!)

Any input appreciated!

---

<div class="post-metadata">

**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\
**Post date:** [February 21, 2016, 10:33pm UTC](https://discuss.elastic.co/t/using-repository-hdfs-on-different-yarn-cluster/41414/2 "2016-02-21T22:33:32Z")

</div>

Moving forward the plugin will work with Hadoop libraries that are embedded/known in the lib folder. And the reason for this being security - delegating to the classpath means we don't know the code source, that is what libraries are used as oppose to the plugin / embedded approach where their location is clearly known.

So while it is a bit more work, selecting the jars and putting them in the plugin (symlink or mounted path can alleviate this problem) folder is the solution regardless of the Hadoop flavor/platform.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:26pm UTC](https://discuss.elastic.co/t/using-repository-hdfs-on-different-yarn-cluster/41414/3 "2017-07-06T13:26:07Z")

</div>