How to do bulk insert from Hive to Elasticsearch?

Elastic Stack Elasticsearch
1

I know that there is a bulk API in ES which can load data from a file, stating that file should have metadata/action before actual data for all the records.

However my use case is to load data from Hive to ES which i'm doing by creating an External Hive table with ES serde. How can I leverage ES bulk load API to load data faster from Hive to ES?

I'm using ES2.1

Thanks!
Ravi

2

Hi,

are you aware of the Hive Integration in ES Hadoop? I'm not too familiar with it so I don't know if this might solve your problem. If not, it would be interesting to know why.

3

Yes, I'm aware of Hive Integration with ES. We are using it to push data successfully to ES from Hive. But the process takes hours. I wanted to know whether bulk load API can improve performance of data load to ES from Hive. If yes, how to use it with Hive? I couldn't find any documentation on that.

4

btw. there is a dedicated sub-forum for Hadoop-related questions like this at https://discuss.elastic.co/c/elasticsearch-and-hadoop, maybe folks there have more of an opinion about this question than here in the Elasticsearch forum.

5

Reposted there.. Thanks!