# How do we store elasticsearch data into Hadoop cluster(Amabri)

**URL:** <https://discuss.elastic.co/t/how-do-we-store-elasticsearch-data-into-hadoop-cluster-amabri/224079>\
**Category:** Elasticsearch\
**Created:** [March 18, 2020, 10:57am UTC](https://discuss.elastic.co/t/how-do-we-store-elasticsearch-data-into-hadoop-cluster-amabri/224079 "2020-03-18T10:57:06Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kishore419](https://avatars.discourse-cdn.com/v4/letter/k/b4bc9f/32.png) [@kishore419](https://discuss.elastic.co/u/kishore419)\
**Post date:** [March 18, 2020, 10:57am UTC](https://discuss.elastic.co/t/how-do-we-store-elasticsearch-data-into-hadoop-cluster-amabri/224079/1 "2020-03-18T10:57:06Z")

</div>

Hello All  
i am using 3 node elasticsearch(7.4.2) cluster in my environment and generating 100GB(appx) data daily, hence i would like to maintain only 30days data in elasticsearch and remaining data should be stored in hadoop(Ambari 2.7.0 - hdfs 3.0.0).  
i found 'Hadoop HDFS Repository' plugin for doing the job  
but could not able to use it.  
can someone please share complete steps for completing the above requirement.

Note: i have installed plugin in all the three elasticsearch nodes  
sudo bin/elasticsearch-plugin install repository-hdfs

unable to understand the below query usage  
PUT \_snapshot/my\_hdfs\_repository  
{  
"type": "hdfs",  
"settings": {  
"uri": "hdfs://namenode:8020/",  
"path": "elasticsearch/repositories/my\_hdfs\_repository",  
"conf.dfs.client.read.shortcircuit": "true"  
}  
}

how to configure path etc value.  
please help me

---

<div class="post-metadata">

**Author:** ![rameshkr1994](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rameshkr1994/32/59029_2.png) [@rameshkr1994](https://discuss.elastic.co/u/rameshkr1994)\
**Post date:** [March 18, 2020, 11:26am UTC](https://discuss.elastic.co/t/how-do-we-store-elasticsearch-data-into-hadoop-cluster-amabri/224079/2 "2020-03-18T11:26:19Z")

</div>

Hi @kishore419.

you want to store elastic search index data into HDFS ?or HDFS Data into elastic search?

```auto
if you want to store data into HDFS then create Hive external table and point on your elastic search index data.then use CTAS to store external hive table into another hive table.

```

`what mean by maintain only 30days data in elasticsearch ` ?

Please explore more ...

Thanks  
HadoopHelp

---

<div class="post-metadata">

**Author:** ![kishore419](https://avatars.discourse-cdn.com/v4/letter/k/b4bc9f/32.png) [@kishore419](https://discuss.elastic.co/u/kishore419)\
**Post date:** [March 19, 2020, 10:22am UTC](https://discuss.elastic.co/t/how-do-we-store-elasticsearch-data-into-hadoop-cluster-amabri/224079/3 "2020-03-19T10:22:27Z")

</div>

> [@rameshkr1994](#):
>
> ```auto
> f you want to store data into HDFS then create Hive external table and point on your elastic search index data.then use CTAS to store external hive table into another hive table.
> 
> ```
> 
> `what mean by maintain only 30days data in elasticsearch ` ?  
> [kishore]  
> 15+ daily indices are creating in my cluster with 100GB size(total) and i will keep only 30 days indices data in elasticsearch and move the older data to Hadoop cluster
> 
> Please explore more ...

hi Ramesh  
i want to store Elasticsearch data to Hadoop  
i have found 'Hadoop HDFS Repository Plugin' for doing this job  
but i was stuck while executing the following command from kibana(dev tools)

PUT \_snapshot/my\_hdfs\_repository  
{  
"type": "hdfs",  
"settings": {  
"uri": "hdfs://namenode:8020/",  
"path": "elasticsearch/repositories/my\_hdfs\_repository",  
"conf.dfs.client.read.shortcircuit": "true"  
}  
}

i am new to hadoop and using hadoop 3 node cluster using ambari.  
do we need to create any path in hadoop cluster  
or suggest hadoop configuration changes.

let me know if you have any questions.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 16, 2020, 10:23am UTC](https://discuss.elastic.co/t/how-do-we-store-elasticsearch-data-into-hadoop-cluster-amabri/224079/4 "2020-04-16T10:23:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.