Create index to HDFS from ElasticSearch


(Roberto Sancho Rojas) #1

Hi:

I want to index the data from HDFS and show to the kibana.
Actaully elastic and kibana are working in locally directory, but can I search from HDFS JSON files?? How??

Thanks


(Dylan Johnson) #2

https://www.elastic.co/products/hadoop


(Roberto Sancho Rojas) #3

Thanks but what i need is store the data and index there like a gateway, but i have readed its deprecated, so i dont need a snapshop or restore.

Wich plugin and configuration i need??

Many thanks


(David Pilato) #4

You need to write a process which will read your data from HDFS, transform to JSON then send to elasticsearch.
Or you can use es-hadoop to do it.

Data needs to be indexed into elasticsearch before you can use Kibana.

I'd suggest looking at this video: https://www.elastic.co/webinars/elasticsearch-and-apache-hadoop

Might help.


(Roberto Sancho Rojas) #5

Hi:
Actually i need index the data from HDFS and store the data and index also into HDFS, is posible??? i dont want store any file into local filesystem.

I read Gateway but is just to store metadata.

Thanks


(David Pilato) #6

You could may be mount an HDFS file system on your machine and put your indices in it but it's absolutely not recommended!


(Mark Walkom) #7

Yeah I'd definitely stay away from that.


(system) #8