Value got nulled when ingesting to ES from Hadoop using Spark

my env:

cloudera hadoop 6.3
es 7.10
spark 2.4.0
elasticsearch-spark-20_2.11-7.10.0.jar

my code very simple:

import org.elasticsearch.spark.rdd.api.java.JavaEsSpark;

JavaEsSpark.saveToEs(rdd_tJava_1, "nyc_yellow_taxi/docs");

the output is as expected, the record count is the same. but when inspecting the records, there are problems, some columns got nulled. all of the records from that colums.

so columns:

pulocationid, vendorid, dolocationid, dolocationid

got nulled.

example here a record in hadoop:

and here the output in elastic:

any idea guys?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.