We created the Hive external table using ElasticSearch StorageHandler as shown below:
CREATE EXTERNAL TABLE DEFAULT.ES_TEST ( REG_DATE STRING , STR1 STRING , STR2 STRING , STR3 STRING , STR4 STRING , STR5 STRING ) ROW FORMAT SERDE 'org.elasticsearch.hadoop.hive.EsSerDe' STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES ( 'es.resource' = 'log-22-20210120' , 'es.nodes' = '126.96.36.199' , 'es.port' = '9201' , 'es.mapping.date.rich' = 'false' );
And then we tried to load the ES data into Hive managed table as like:
insert overwrite table elastic.es_log_tab partition(part_log_date) select * , current_timestamp() , from_unixtime(unix_timestamp(reg_date), 'yyyyMMdd') from DEFAULT.ES_TEST;
When the ES data for given date is volumed to about 65GB, it was approximated taken 10 hours or 1.1M rows per minute (670M rows in total).
In order to get a better loading performance for this case, are there any further recommendation or checkpoints? How about increasing the number of mappers? Currently, it is running with 16 mappers. Expecting to get it faster with more mappers?
Please share your thoughts and previous experience with me.