Create index to HDFS from ElasticSearch

Roberto_Sancho_Rojas · February 8, 2016, 9:51pm

Hi:

I want to index the data from HDFS and show to the kibana.
Actaully elastic and kibana are working in locally directory, but can I search from HDFS JSON files?? How??

Thanks

Dylan_Johnson1 · February 9, 2016, 3:47am

https://www.elastic.co/products/hadoop

Roberto_Sancho_Rojas · February 9, 2016, 6:12am

Thanks but what i need is store the data and index there like a gateway, but i have readed its deprecated, so i dont need a snapshop or restore.

Wich plugin and configuration i need??

Many thanks

dadoonet · February 9, 2016, 6:55am

You need to write a process which will read your data from HDFS, transform to JSON then send to elasticsearch.
Or you can use es-hadoop to do it.

Data needs to be indexed into elasticsearch before you can use Kibana.

I'd suggest looking at this video: https://www.elastic.co/webinars/elasticsearch-and-apache-hadoop

Might help.

Roberto_Sancho_Rojas · February 9, 2016, 7:28am

Hi:
Actually i need index the data from HDFS and store the data and index also into HDFS, is posible??? i dont want store any file into local filesystem.

I read Gateway but is just to store metadata.

Thanks

dadoonet · February 9, 2016, 7:42am

You could may be mount an HDFS file system on your machine and put your indices in it but it's absolutely not recommended!

warkolm · February 10, 2016, 1:14am

Yeah I'd definitely stay away from that.