# Org.elsticsearch -- trying to run scripted upserts, written in painless language, from Spark -- {type=null\_pointer\_exception, reason=null}

**URL:** <https://discuss.elastic.co/t/org-elsticsearch-trying-to-run-scripted-upserts-written-in-painless-language-from-spark-type-null-pointer-exception-reason-null/238968>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [June 27, 2020, 6:14pm UTC](https://discuss.elastic.co/t/org-elsticsearch-trying-to-run-scripted-upserts-written-in-painless-language-from-spark-type-null-pointer-exception-reason-null/238968 "2020-06-27T18:14:09Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Eman\_Mughal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eman_mughal/32/71248_2.png) [@Eman\_Mughal](https://discuss.elastic.co/u/Eman_Mughal)\
**Post date:** [June 27, 2020, 6:14pm UTC](https://discuss.elastic.co/t/org-elsticsearch-trying-to-run-scripted-upserts-written-in-painless-language-from-spark-type-null-pointer-exception-reason-null/238968/1 "2020-06-27T18:14:09Z")

</div>

Hi There,

I am trying to run the following code

```auto
df.write
  .format("org.elasticsearch.spark.sql")
  .option("es.nodes.wan.only","true")
 // .option("es.nodes.discovery", "false")
 // .option("es.nodes.client.only", "false")
  .option("es.mapping.id", "fileName")
  .option("es.resource","<myIndex>/_doc")
  .option("es.update.script.stored","mp-upsert-with-dup-check-and-revisions")
  .option("es.script.lang","painless")
  .option("es.write.operation", "upsert")
  .option("es.nodes", esURL)
  .mode("append")
  .save("<myIndex>/_doc")

```

and getting following error

```auto
org.elasticsearch.hadoop.rest.EsHadoopRemoteException: illegal_argument_exception: failed to execute script;org.elasticsearch.hadoop.rest.EsHadoopRemoteException: script_exception: runtime error;org.elasticsearch.hadoop.rest.EsHadoopRemoteException: 
{type=null_pointer_exception, reason=null}

```

My Painless Code is

```auto
   {
      "_id" : "mp-upsert-with-dup-check-and-revisions",
      "found" : true,
      "script" : {
        "lang" : "painless",
        "source" : """
                    // Get Old and New Documents
                    def old_source = new HashMap(ctx._source);
                    def new_source = new HashMap(params.newsource);
                    

                    // Duplicate Check
                    def pure_old_source = new HashMap(ctx._source);
                    pure_old_source.remove("ingestion_meta");
                    pure_old_source.remove("_revisions");
                    
                    def pure_new_source = new HashMap(params.newsource);
                    pure_new_source.remove("ingestion_meta");
                    
                    
                    // Noop Check and Revision Tracking
                    if(pure_old_source == pure_new_source){
                      ctx.op = 'none';
                    }else{
                      if(old_source.ingestion_meta != null){
                        if(old_source._revisions == null){
                          new_source._revisions = [];
                        }else{
                          new_source._revisions = new ArrayList(old_source._revisions);
                          old_source.remove("_revisions")
                        }
                        new_source._revisions.add(old_source);
                      }
                      
                    }

                    // Add Ingestion_Meta if missing
                    if(new_source.ingestion_meta == null){
                        new_source.ingestion_meta = new HashMap();
                    }
                    
                    
                    // Add created_at timestamp
                    if(old_source.ingestion_meta == null){
                      new_source.ingestion_meta.created_at = ctx._now;
                    }else{
                      new_source.ingestion_meta.created_at = old_source.ingestion_meta.created_at;
                    }
                    
                    // Add modified_at timestamp
                    new_source.ingestion_meta.modified_at = ctx._now;
                    
                    ctx._source = new_source;

                  """
      }
    }

```

---

<div class="post-metadata">

**Author:** ![Eman\_Mughal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eman_mughal/32/71248_2.png) [@Eman\_Mughal](https://discuss.elastic.co/u/Eman_Mughal)\
**Post date:** [June 28, 2020, 7:34pm UTC](https://discuss.elastic.co/t/org-elsticsearch-trying-to-run-scripted-upserts-written-in-painless-language-from-spark-type-null-pointer-exception-reason-null/238968/2 "2020-06-28T19:34:59Z")

</div>

I think this feature has still not been added to es.hadoop package.

Written on this URL

> <https://stackoverflow.com/questions/60810889/scripted-upsert-with-elasticsearch-hadoop-impossible>

moving on, it is also the last comment on this URL

> <https://github.com/elastic/elasticsearch-hadoop/issues/538>
>
> Scripting has added new options such as id, file or inline. This should be supported as well by the connector.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 26, 2020, 7:35pm UTC](https://discuss.elastic.co/t/org-elsticsearch-trying-to-run-scripted-upserts-written-in-painless-language-from-spark-type-null-pointer-exception-reason-null/238968/3 "2020-07-26T19:35:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.