I would like to read elasticsearch data into Hive, and have been following documentation here.
One of my elasticsearch fields (ACTIVITIES) is an array of JSON objects with structure like this:
ACTIVITIES:{
{NAME: "..."
ID: "..."
TIME: "..."
},
{NAME: "..."
ID: "..."
TIME: "..."
},
.....
}
Here is the mapping of that field:
"ACTIVITIES" : {
"properties" : {
"NAME" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword"
}
}
},
"ID" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword"
}
}
},
"TIME" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword"
}
}
}
}
},
I want to map this into Hive column(s). I tried to create the column as an array of struct array<struct<name:string,id:string,time:string>>
and give mapping as
'es.mapping.names' : 'activities.name:ACTIVITIES.NAME, activities.id:ACTIVITIES.ID, activities.time:ACTIVITIES.TIME '
All other fields from ES are read correctly into Hive except this JSON array, which is read as NULL. I dont know right way to do, because I couldn't find type conversion for JSON array like this in documentation.
Please help