I'm using spark streaming and update ES by append objects to some nested type fields.
Currently I'm query a doc from Es, then construct some map objects and update the doc by append the objects to array of nested type field, and finally upsert the doc to ES, Is there a better solution?
UPDATE:
I think java scripted update api would be better.
Hello,
being in the same situation. When trying to save document with nested object I've this kind on error
16/04/25 15:53:30 ERROR Executor: Exception in task 0.0 in stage 116.0 (TID 116) org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: [ScalaMapFieldExtractor for field [[rowKey]]] cannot extract value from entity [class java.lang.String] | instance ... at org.elasticsearch.hadoop.serialization.bulk.AbstractBulkFactory$FieldWriter.write(AbstractBulkFactory.java:97) at org.elasticsearch.hadoop.serialization.bulk.TemplatedBulk.writeTemplate(TemplatedBulk.java:80) at org.elasticsearch.hadoop.serialization.bulk.TemplatedBulk.write(TemplatedBulk.java:56) at org.elasticsearch.hadoop.rest.RestRepository.writeToIndex(RestRepository.java:163) at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:49) at org.elasticsearch.spark.rdd.EsSpark$$anonfun$saveToEs$1.apply(EsSpark.scala:67) at org.elasticsearch.spark.rdd.EsSpark$$anonfun$saveToEs$1.apply(EsSpark.scala:67) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:88) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 16/04/25 15:53:30 WARN ThrowableSerializationWrapper: Task exception could not be deserialized java.lang.ClassNotFoundException: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
1 Like