I'm having an issue very similar to this one, but I'm not sure exactly what
they did to get the array contents. I can't apply the toStrings() function
to the arraywritable because it's in a writable.
http://grokbase.com/t/gg/elasticsearch/14c2sb14rk/hadoop-getting-specified-fields-from-elasticsearch-as-an-input-to-a-mapreduce-job/14c3dz4san#14c3dz4san
I've got a map from document IDs to another map from fields to their
contents. I'm interested in getting the contents of the concepts field,
which is an array of tuples. My goal is to use this in a recommender
system using spark. I'm working in scala.
Looking up a document ID gives me a map from the field name concepts to the
contents. I want the contents of the concepts field in the form of a map
from text to relevance. I'd rather not save this as a string and parse it
for efficiency reasons.
scala> map("396974")
warning: there were 1 deprecation warning(s); re-run with -deprecation for
detai
ls
res34: org.apache.hadoop.io.MapWritable = {concepts=[{text=Jujutsu,
relevance=0.
953505}, {text=Aikido, relevance=0.87238}, {text=Martial arts,
relevance=0.56092
8}, {text=Morihei Ueshiba, relevance=0.535638}, {text=Mixed martial arts,
releva
nce=0.470634}, {text=Grappling, relevance=0.43726}, {text=Karate,
relevance=0.43
5142}, {text=Brazilian Jiu-Jitsu, relevance=0.403991}]}
scala> map("396974").get(new Text("concepts"))
warning: there were 1 deprecation warning(s); re-run with -deprecation for
detai
ls
res35: org.apache.hadoop.io.Writable =
org.elasticsearch.hadoop.mr.WritableArray
Writable@1077a7
My understanding is that this is a writable wrapper around an
arraywritable object. I want to get the contents of this object as a map.
Ultimately, I want is a map from document IDs to the corresponding concept
map from text to relevance. The thread linked above suggests calling
toStrings on the writablearraywritable object, but this doesn't work:
scala> map("396974").get(new Text("concepts")).toStrings
:41: error: value toStrings is not a member of
org.apache.hadoop.io.Wri
table
map("396974").get(new Text("concepts")).toStrings
^
Any advice will be greatly appreciated.
Thanks,
Paul
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/239004a9-81ed-4aef-9a19-35a64655fa50%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.