Hive to ES incremental upload

(Ashwini Kumar) #1

I have 20000 documents in ES and I have uploaded them from Hive using Hive -ES connector. How can I dump new data from hive to ES? I want to do an incremental upload how sqoop does.

(James Baiera) #2

@Ashwini_Kumar I'm not aware of anything that does this out of the box for Hive, so it might require a little bit of footwork to do. Does your new data have any way to differentiate it from old data such as a create timestamp or transaction ids? If so, you could attempt to query just for the new data since your last load into ES when running the load command.

(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.