Add Column at runtime through inline scipt : Date difference

Elastic Stack Elasticsearch
1

I have a situation, where i need to calculate a value based on if the column is present and add this to all document and it should be indexed.

How can I add a column to my indexed document by using doc. ctx and fields are not working. What are other api that a doc have

I have following script :
SCRIPT

GET  /scm_master/suborder_details/_search?
{
  "query": {
    "filtered": {
      "filter": {
        "bool": {
          "must": [
             {"exists": { "field": "created_on" }},
             { "missing": {"field": "new_field"}
            }
          ]
        }
      }
    }
  },
   "script_fields" : {
     "new_field" : {
       "script" :"def duration = groovy.time.TimeCategory.minus(new java.util.Date(doc['updated_on'].value), new java.util.Date(doc['created_on'].value));;"
     }
  }
}

output

{
  "took": 7,
  "timed_out": false,
  "_shards": {
    "total": 10,
    "successful": 10,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 1,
    "hits": [
      {
        "_index": "scm_master",
        "_type": "suborder_details",
        "_id": "subOrderCode5",
        "_score": 1,
        "fields": {
          "new_field": [
            "0.207 seconds"
          ]
        }
      }
    ]
  }
}

it runs successfully, but a new field is not created.

2

script_fields just adds the field to the search result which it looks like it did properly. I think what you are looking for is update by query which is a feature coming in 2.3 and 5.0.

3

Thanks Nik.

Is there any solution available for 2.2, any hack :frowning:

When should we expect 2.3.

Also I have one query : even if we update a document via script is it that the old document is deleted and updated document is re indexed like fresh. And so updating a doc is more time taking?

4

Use the update api on all of the documents you want to change? Update API | Elasticsearch Guide [8.11] | Elastic

Updates are basically atomic deletes and adds. Using the script just saves you a round trip back to the client.

A couple of weeks I think.

5

Thanks Again Nik.

Yes at this moment, I am using update api only for all documents. But I am having an issue.

When I am updating this document via below script, then at same time may be there is another update at that index via java update api.

Issue : sometimes i see the new fields get created but some times not, but if i re run the script, it gets created . And we run query only well the condition written in script meets. (But still we have if condition in script to be double sure). Is it due to the case that may be at same time some other thread trying to update the same document ?

I run below script using java bulk processor api.

//my script

if( ctx._source.containsKey('order_created_date') && ctx._source.containsKey('sp_awb_uploaded_status_date') ){def duration = groovy.time.TimeCategory.minus(new java.util.Date("5237683"), new java.util.Date("12321223")); ctx._source.order_to_shipped = duration.days;}

//My Java function

for (ElasticDataDTO dto : dtos) {
try {
UpdateRequest updatReq = new UpdateRequest(indexName, indexType, dto.getSubOrderCode());
if(null==params)
params= new HashMap<String, Object>();
Script scriptObj = new Script(script, ScriptType.INLINE, null,params);
updatReq.script(scriptObj);
bulkProcessor.add(updatReq);
}catch (Exception e) {
throw new RuntimeException(e);
}
}

6

You should wrap code blocks in ``` so they are readable, please.

You should check the responses on those update requests - if you see any version conflicts then you know that an in flight conflict caused the trouble. There is a retry_on_conflict parameter you can use to retry the script.

7

Thanks Nik.

I will try the retry_on_conflict.