How do I storage ES data into HDFS

Hello guys, I'm using ElasticSearch 6.6.0. Recently I recieved a requirement that ES data should storage into HDFS, because Solr can do this.
So I mounted HDFS as a local path '/esdata' by HDFS NFS3 Gateway, and set data path to this, but when I started a single node ElasticSearch and created an empty index 'test_idx' without mapping, it failed, and threw exception as below:

org.elasticsearch.indices.recovery.RecoveryFailedException: [test_idx][1]: Recovery failed on {node-1}{...}{...}{ip}{ip:port}{ml.machine_memory=8202268368, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
...
...
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed to recover from gateway
...
...
Caused by: java.io.IOException: Invalid argument

These 2 discussion below shows that ES could use HDFS for storage in costin's opinion(one member of ElasticSearch):
https://discuss.elastic.co/t/can-elasticsearch-reads-and-stores-data-in-hdfs-by-es-hadoop/40713/2
https://github.com/elastic/elasticsearch/issues/9072

I don't know how to fix this. It shows that "File append is supported but random write is not supported." in hadoop official site's partition "HDFS NFS Gateway".
I have tried some tests after mounting HDFS, the mounting path is "/esdata":
(1) echo 111 > /esdata/1.log # success
(2) touch /esdata/1.log # success
echo 111 > /esdata/1.log # success
(3) echo 111 > /esdata/1.log # success
echo 111 > /esdata/1.log # failed, do again
(4) echo 111 > /esdata/1.log # success
echo 111 >> /esdata/1.log # success, do again
(5) echo 111 >> /esdata/1.log # success
echo 111 >> /esdata/1.log # success, do again

Welcome to our community! :smiley: We aren't all guys though,

Please be aware that we do not recommend or support running on HDFS. Running a distributed system on a distributed filesystem is not a great idea.

Thanks for your reply.
But Solr can use HDFS to storage data.
Is there any plan to support ES storage data into HDFS in the future?

Not officially, no.

alright, thanks.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.