We are adding documents through a pyspark dataframe to elastic.
df.write.mode('append').format('org.elasticsearch.spark.sql')
.option('es.nodes', 'https://xyz.com')
.option('es.net.http.auth.user','userid')
.option('es.net.http.auth.pass','pwd')
.option('es.net.ssl', 'true')
.option('es.nodes.wan.only', 'true')
.option('es.net.ssl.cert.allow.self.signed','true')
.option('es.net.ssl.truststore.location', 'file:///opt/escerts/test.jks')
.option('es.net.ssl.keystore.location','file:///opt/escerts/test.jks')
.option('es.net.ssl.truststore.pass', 'password')
.option('es.net.ssl.keystore.pass','password')
.option('es.resource', 'index_name/_doc')
.option('es.mapping.id','response.firstcolumn')
.option('es.batch.size.bytes', '70mb')
.option('es.batch.size.entries', '23000')
.option('es.batch.write.refresh', 'false').save()
Earlier I tried with .option('es.resource', 'index_name') and got this error. So I modified to 'index_name/_doc' and error gone.
org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: No type found; Types are required when writing in ES versions 6 and below. Expected [index]/[type], but got [index_name]
Finally, it ended up with this error. In the previous version, it worked fine. This happens after an upgrade. Index id is coming with separate node, which is causing issue. When tried it in kibana by adding this doc without {"index":{"id":"3J000499841"}}, it is added without any issue. Please help me to solve this issue.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 4) (ip-10-92-229-81.ec2.internal executor 1): org.apache.spark.util.TaskCompletionListenerException: org.elasticsearch.hadoop.rest.EsHadoopRemoteException: mapper_parsing_exception: failed to parse
{"index":{"id":"3J000499841"}}
{"response":{"a_est_clm_val":0,"a