Hive to elastic search Data loading...Index status = RED


(Siva Sai) #1

when i load data from hive to elastic search, first time i got only 10 millions of records from 160 millions. when loaded for the second time i am not able to get any records and elastic search index status is RED.

showing below error


INSERT OVERWRITE TABLE rtt_analysis_ext7 select rt.procedure_number, rt.pg w_name, rt.procedure_start_time, rt.procedure_end_time, rt.asr5k_callid, rt.tai_ mcc_mnc_tac, rt.cell_id, rt.asr5k_disconnect_reason,rt.gtp_cause_value,rt.diamet er_termination_cause, rt.gtp_cause_number from rtt_analysis rt;
Query ID = root_20161024042635_8d449250-ca52-4bdd-9486-5fb262ded953
Total jobs = 1
Launching Job 1 out of 1

Status: Running (Executing on YARN cluster with App id application_1476839126138 _0088)


    VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED

Map 1 FAILED 47 0 0 47 84 0

VERTICES: 00/01 [>>--------------------------] 0% ELAPSED TIME: 2404.18 s

Status: Failed
Vertex failed, vertexName=Map 1, vertexId=vertex_1476839126138_0088_1_00, diagno stics=[Task failed, taskId=task_1476839126138_0088_1_00_000034, diagnostics=[Tas kAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeExce ption: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveExcept ion: Hive Runtime Error while processing row {"pgw_rtt_record_number":null,"pgw_ rtt_version_number":null,"procedure_number":3,"pgw_name":"CHRXNCLHPNR","procedur e_start_time":"2016-01-27 00:34:35.293","procedure_end_time":"2016-01-27 00:34:3 5.293","asr5k_callid":1756141029,"imsi":null,"msisdn":null,"imeisv":null,"tai_mc c_mnc_tac":"311-480-0x9f13","cell_id":"0x26f2e02","asr5k_disconnect_reason":2,"s erving_network":null,"radio_access_technology":null,"handover_indicator":null,"s gw_hsgw_teid":null,"pgw_teid":null,"an_gw_address":null,"access_point_name":null ,"framed_ip_address":null,"framed_ipv6_address":null,"uplink_ambr":null,"downlin k_ambr":null,"pco_dns_ipv6_address_primary":null,"pco_dns_ipv6_address_secondary ":null,"pco_dns_ipv4_address_primary":null,"pco_dns_ipv4_address_secondary":null ,"eps_bearer_id":null,"linked_bearer_identity":null,"uplink_mbr":null,"downlink_ mbr":null,"uplink_gbr":null,"downlink_gbr":null,"gtp_cause_value":"16","piggybac k_record_indicator":null,"session_id":null,"auth_application_id":null,"pgw_host_ origin_host":null,"diameter_peer_address_realm":null,"dest_peer_host":null,"cc_r equest_type_":null,"cc_request_number":null,"result_code":null,"origin_state_id" :null,"service_selection":null,"charging_gateway_function_host":null,"charging_g roup_id":null,"server_name_cscf_address":null,"framed_pool":null,"framed_ipv6_po ol":null,"auth_request_type":null,"re_auth_request_type":null,"diameter_terminat ion_cause":null,"qos_class_identifier":null,"ip_can_type":null,"event_trigger2": null,"event_trigger":null,"charging_rule_remove":null,"charging_rule_install_":n ull,"multiple_services_indicator":null,"multiple_services_credit_control_rating_ group":null,"multiple_services_credit_control_granted_service_unit":null,"proces s_date":null,"process_time":null,"process_duration":null,"process_15mints":null, "process_30mints":null,"process_hour":null,"gtp_cause_number":"16"}
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProce ssor(TezProcessor.java:171)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java :137)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOPro cessorRuntimeTask.java:344)


(Jimmy Kuang) #2

What happens when load the data directly into a local Hive table and not into ES?


(system) #3