HDFS as elastic search data repository

Elastic Stack Elasticsearch
#1

For large volume of logs we are thinking of using HDFS as data repository for elastic search.
apache flume is one of option suggested by few blogs on internet. Would like to get more information from community on this to come on conclusion. How do I instruct Elastic search to store data in HDFS and query/index same every time.

LOGSTASH ---> ELASTIC SEARCH ( want to use Kibana for visualization ) -- > HDFS

#2

Except for ES-Hadoop I don't believe HDFS is a supported storage backend for Elasticsearch.

#3

It is not.

#4

Thanks @magnusbaeck. ES-hadoop as far as I understood from documentation is to do search on Hadoop echo system. Where as my requirement is to just use high storage capacity of Hadoop.

I will be collecting huge logs from 'n' number of Micro services and need to store this in faster storage for better retrieval. any thought on that ?.

I am getting tilt towards mongoDb a bit, doing research on elastic and mongodb for the moment

#5

any link/guide to setup elastic indices in Hadoop?

#6

I don't think it is supported... if you want it, you need to build one yourself... do it for the community :wink:

Here is an idea that you can think about, check out the link below

#7

You should really not do this. Really...

That being said, may be you can mount an NFS drive running on hadoop and put your indices in it but again, I would not do it...

Why not sending your logs directly to elasticsearch?

#8