Object Array in Painless source loses values and indexes with "[]" for values

[jthoni (John T)]

We have just upgraded from ES/Nest v5 to v 6.8. We have an api that does an upsert using a Painless script. The script checks to see that the lastUpdated date in the payload is greater than that in the indexed document, then it upserts the fields.

I have debugged, and I see that the items in the params Dictionary are correct in the Inline object. We create it with the following, with dictData being the Dictionary of properties on the doc.

return new InlineScript(script)
{
      Lang = "painless",
      Params = dictData,
 };

We then create an UpdateRequest:

return new UpdateRequest<object, object>(FeedbackParentIndexAlias, FeedbackParentMainType, "feedbackparent|" + parentData.ParentFeedbackId.ToString())
{ 
    RetryOnConflict = 1,
    DetectNoop = true,
    Upsert = lowerCaseParent,
    Script = inlineScript,
    Refresh = triggerRefresh ? Refresh.True : Refresh.False,
};

We then send this with:
var updateResponse = await this._nestClient.UpdateAsync<object, object>(upsertQuery, CancellationToken.None);

If I look at upsertQuery in the debugger, or I write to a string with either:

this._nestClient.SourceSerializer.SerializeToString(upsertQuery)

or

JsonConvert.SerializeObject(upsertQuery, Newtonsoft.Json.Formatting.None)

It shows the values for the properties inside of the engineeringwork array

{
    "script": {
      "inline": "def sf = new SimpleDateFormat(\"yyyy-MM-dd'T'HH:mm:ss.SSS\");if(ctx._source.triagelastupdated != null && sf.parse(ctx._source.triagelastupdated).after(sf.parse(params.datetimecomparison))) { ctx.op = \"noop\"; } else {ctx._source.engineeringwork = params.engineeringwork;}",
      "lang": "painless",
      "params": {
        "datetimecomparison": "2021-09-14T03:03:28.017Z",
        "engineeringwork": [
          {
            "exampleInteger": 12345,
            "exampleLink": "https://foo.com/id.456",
            "exampleTitle": "This is an example"
          }
        ],        
        "triagelastupdated": "2021-09-14T03:03:28.017Z"
      },
      "source": "def sf = new SimpleDateFormat(\"yyyy-MM-dd'T'HH:mm:ss.SSS\");if(ctx._source.triagelastupdated != null && sf.parse(ctx._source.triagelastupdated).after(sf.parse(params.datetimecomparison))) { ctx.op = \"noop\"; } else {ctx._source.engineeringwork = params.engineeringwork;}"
    },
    "upsert": {
      "This is": "The doc that would get indexed if there was no existing one"
    }
  }

When the upsert completes, the value indexed for engineeringwork is:

        "engineeringwork": [
          {
            "exampleInteger": [],
            "exampleLink": [],
            "exampleTitle": []
          }

Importantly, if I look at the Request that is returned back from the cluster shows that the Request sent was:

{
    "script": {
      "inline": "def sf = new SimpleDateFormat(\"yyyy-MM-dd'T'HH:mm:ss.SSS\");if(ctx._source.triagelastupdated != null && sf.parse(ctx._source.triagelastupdated).after(sf.parse(params.datetimecomparison))) { ctx.op = \"noop\"; } else {ctx._source.engineeringwork = params.engineeringwork;}",
      "lang": "painless",
      "params": {
        "datetimecomparison": "2021-09-14T03:03:28.017Z",
        "engineeringwork": [
          {
            "exampleInteger": [],
            "exampleLink": [],
            "exampleTitle": []
          }
        ],        
        "triagelastupdated": "2021-09-14T03:03:28.017Z"
      },
      "source": "def sf = new SimpleDateFormat(\"yyyy-MM-dd'T'HH:mm:ss.SSS\");if(ctx._source.triagelastupdated != null && sf.parse(ctx._source.triagelastupdated).after(sf.parse(params.datetimecomparison))) { ctx.op = \"noop\"; } else {ctx._source.engineeringwork = params.engineeringwork;}"
    },
    "upsert": {
      "This is": "The doc that would get indexed if there was no existing one"
    }
  }

If I take the string representation (i.e. the serialized query) and run it from Kibana, it updates correctly.

It seems that params Dictionary that is being sent by the Nest client is not parsing the values correctly. I do always get the correct number of items, and each item has the correct property names, it is just the values that are missing.

The mapping for that field, is:

"engineeringwork" : {
            "type" : "nested",
            "include_in_parent" : true,
            "include_in_all" : false,
            "properties" : {
              "exampleTitle" : {
                "type" : "text",
                "analyzer" : "case_insensitive",
                "fielddata" : true
              },
                "exampleLink" : {
                "type" : "text",
                "analyzer" : "case_insensitive",
                "fielddata" : true
              },
              "exampleInteger" : {
                "type" : "integer"
              }
            }
          }

BTW, I am using JsonNetSerializer on my Nest client:

ConnectionSettings settings = new ConnectionSettings(pool, new SecureElasticConnection(), sourceSerializer: JsonNetSerializer.Default);

Any help would be appreciated.

Thanks,
~john

[jthoni (John T)]

I just noticed in Fiddler that the outgoing object only has the keys and not the values. So the dictionary set in the UpdateRequest has all values, but what gets sent is:

[jthoni (John T)]

It seems that when the dictionary is serialized from the Params, it is using the RequestResponseSerializer. That does not seem to be able to parse the values from an object, only the keys. Here is a stand alone example:

static void Main(string[] args)
        {
            var node = new Uri("http://fake");
            var pool = new SingleNodeConnectionPool(node);
            ConnectionSettings settings = new ConnectionSettings(pool, sourceSerializer: JsonNetSerializer.Default);
            settings.DisableDirectStreaming();
            ElasticClient elasticClient = new ElasticClient(settings);

            dynamic dyn = new JObject();
            dyn.First = "This is the first";
            dyn.Num = 1234;

            JArray jar = new JArray();
            jar.Add(dyn);

            var reqRespSerializer = elasticClient.RequestResponseSerializer.SerializeToString(jar);
            Console.WriteLine(reqRespSerializer);

            /*
                Outputs:
                    [
                      {
                        "First": [],
                        "Num": []
                      }
                    ]
            */

            var sourceSerializer = elasticClient.SourceSerializer.SerializeToString(jar);
            Console.WriteLine(sourceSerializer);

            /*
             Outputs:
                [
                  {
                    "First": "This is the first",
                    "Num": 1234
                  }
                ]           
           */

            var jsonConverted = JsonConvert.SerializeObject(jar);
            Console.WriteLine(jsonConverted);
            /*
             Outputs:
               [{"First":"This is the first","Num":1234}]          
           */

            Console.ReadKey();          
        }

A regular JsonConver and the SourceSerializer work, but not RequestResponseSerializer, which seems to be what is being used when sending the request.

[jthoni (John T)]

Well, I have a workaround. Not listing as a solution as it is a hack. Either I need to be defining a custom field resolver or something, or there is a bug in Nest.

So, the SourceSerializer gives me the correct nested object, but other parts of the payload a different and it does not work when sending to ES. What I am doing is:

1. Serialize to json with RequestResponseSerializer and Parse to a JObject
2. Serialize (only the nested object really required) with SourceSerializer to get my object and parse to JObject
3. Remove the bad property from the RequestResponseSerializer object and replace with the good one
4. Use the lowlevel client and pass the good object, parsing to json with JasonConvert.

This works for now.