For large volume of logs we are thinking of using HDFS as data repository for elastic search.
apache flume is one of option suggested by few blogs on internet. Would like to get more information from community on this to come on conclusion. How do I instruct Elastic search to store data in HDFS and query/index same every time.
LOGSTASH ---> ELASTIC SEARCH ( want to use Kibana for visualization ) -- > HDFS