I am trying to use ES hadoop MR to update documents using following properties:
es.write.operation=update
es.mapping.id=_metadata._id
es.read.metadata.field=metadata
es.read.metadata=true
On each document write function call, I am getting this type of exception.
2015-08-20 16:38:34,895 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: [MapWritableFieldExtractor for field [_metadata._id]] cannot extract value from entity [class java.lang.String] | instance [{doc_id=AU9Jtv-wfhVNYAhE2vWb, test_filed=tempData, _metadata={_index=sa-analytics-new-2, _type=logs, _id=AU9KfBg9fhVNYAhE21El, _score=0.0}, Temp_id=AU9KfBg9fhVNYAhE21El}] at org.elasticsearch.hadoop.serialization.bulk.AbstractBulkFactory$FieldWriter.write(AbstractBulkFactory.java:97) at org.elasticsearch.hadoop.serialization.bulk.TemplatedBulk.writeTemplate(TemplatedBulk.java:80) at org.elasticsearch.hadoop.serialization.bulk.TemplatedBulk.write(TemplatedBulk.java:56) at org.elasticsearch.hadoop.rest.RestRepository.writeToIndex(RestRepository.java:148) at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.write(EsOutputFormat.java:151) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:635) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) I think there are some issues in es.mapping.id value. May be it is not allowed to access fields like i have given here. Any help in this will be appreciated Thanks
Can you please format your post? It's just one big block of text
It looks like you are trying to both read and write in the same config which likely means you are confusing the two operations. If you pay attention to the error it explains the issue - you are trying to do an update based on the field metadata.id however it cannot be extracted from the given entity which is of type String.
Are you trying to read the data in JSON format by any chance?
I'm hitting the same "cannot extract entity" error trying to do something similar with
es.mapping.parent=feature.owner.id
Are you saying that this is because I'm trying to write an RDD[String]?
?
@norbert It's best to create a new thread not resurrect one that's so old (2 month). Ideally with your actual configuration and exception plus info on the version of the library being used.