Create index to HDFS from ElasticSearch

Elastic Stack Elasticsearch
1

Hi:

I want to index the data from HDFS and show to the kibana.
Actaully elastic and kibana are working in locally directory, but can I search from HDFS JSON files?? How??

Thanks

2

https://www.elastic.co/products/hadoop

3

Thanks but what i need is store the data and index there like a gateway, but i have readed its deprecated, so i dont need a snapshop or restore.

Wich plugin and configuration i need??

Many thanks

4

You need to write a process which will read your data from HDFS, transform to JSON then send to elasticsearch.
Or you can use es-hadoop to do it.

Data needs to be indexed into elasticsearch before you can use Kibana.

I'd suggest looking at this video: https://www.elastic.co/webinars/elasticsearch-and-apache-hadoop

Might help.

5

Hi:
Actually i need index the data from HDFS and store the data and index also into HDFS, is posible??? i dont want store any file into local filesystem.

I read Gateway but is just to store metadata.

Thanks

6

You could may be mount an HDFS file system on your machine and put your indices in it but it's absolutely not recommended!

1 Like
7

Yeah I'd definitely stay away from that.

1 Like