HDFS to Elastic search - User Triggered Data Load

Elastic Stack Elasticsearch
1

Hi All

We are using flume, sqoop, hadoop, elastic-search and kibana to perform real time analytic. However we are wondering how to dynamically query/retrieve data's in hdfs from elastic-search via kibana. Currently we have pushed the data's from hadoop to elastic-search using map-reduce. But how can we pull the data from hdfs from ES via kibana?

The intention was to use Hadoop to persist longer data (60 days) and ES to store for the last 3 days of data with this as a use case how can we retrieve data older than 3 days?

Any help would be helpful in progressing further.

Regards
Rinu

2

Hi All,

Any pointers to this would be really helpful.

Regards
Rinu