Hi,recently I need to integrate ES and hadoop via elasticsearch-hadoop, What about the performance of reading data from ES to hadoop, and the performance of reading data from hadoop to ES？Let‘s say that I have 100G data per day need to process. What should i chose? 1.data-hadoop-ES. or 2.data- ES-hadoop.
THANKS
@JIELOGAN, is your question about which version of Hadoop to use, v1 or v2?
If so, that's not an ES-Hadoop question as the answer depends on which version of Hadoop you are running, 1.x or 2.x. If you're starting new with Hadoop, I'd definitely recommend going with the most current version, 2.8 (assuming you're not limited to a given version via some other requirement)
With regards to write performance from Hadoop into Elasticsearch, see: https://www.elastic.co/guide/en/elasticsearch/hadoop/current/arch.html. In short, the level of parallelism for the writing is based on the number of splits and mappers in your dataset/cluster: more splits == more mappers == more parallelism
@JIELOGAN I would use Hadoop 2.7.x if you are planning on using ES-Hadoop. 2.8 is not yet supported.
Doh, thanks for the clarification, @james.baiera!
thank you for your help
thanks, i also used 2.7.x
Hi ,
i have large amount of data in ES (2.4.1) index .i need the index data moved into hadoop hdfs stroage. how to do the task .
following steps iam doing
Es version 2.4.1
Apache hadoop version 2.7.3
/elasticsearch-2.4.1$ bin/plugin install elasticsearch/elasticsearch-repository-hdfs/2.4.1
Start the elasticsearch
start hadoop
after what iam doing ???
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.