"Key" and "docCount" are missing when serializing BucketAggregate to string with Nest client

[jthoni (John T)]

I am working on upgrading a service from ES 5.0 to 6.8. I have a bucket aggregate that in v5 serializes to this:

      "items": [
        {
          "key": "random+topic",
          "docCount": 27919,
          "aggregations": {
            "ParentReference": {
              "docCount": 24992,
              "aggregations": {
                "Popularity": {
                  "value": 25223
                }
              }
            }
          }
        },
        {
          "key": "unknown problem+latency",
          "docCount": 24566,
          "aggregations": {
            "ParentReference": {
              "docCount": 23419,
              "aggregations": {
                "Popularity": {
                  "value": 23931
                }
              }
            }
          }
        },

With the v6 of Elasticsearch.Net and Nest, when serialized, I end up with:

      "items": [
        {
          "ParentReference": {
            "Popularity": {
              "value": 25223
            }
          }
        },
        {
          "ParentReference": {
            "Popularity": {
              "value": 23931
            }
          }
        },

I had previously encountered the issue where the "aggregations" property is no longer returned (though I would love a pointer to the breaking changes announcement on that), and have updated my code accordingly. I can't do much without the Key and docCount, however. I figure there must be something related to the Json parsing changes.

I have already tried the steps in:
Custom Serialization | Elasticsearch.Net and NEST: the .NET clients [6.x] | Elastic

I have tried with the default serializer, as well as a custom one using the JsonNetSerializer.Default to no effect.

Can anyone provide a suggestion on what I should be doing?

note that this is how I am getting my BucketAggregate:

        var childAgg = response.Aggregations[ss.Type] as SingleBucketAggregate;
        var nestedAgg = childAgg.Aggregations[ss.Path] as SingleBucketAggregate;
        var countAgg = nestedAgg.Aggregations[ssTermsName] as BucketAggregate;
        return new ProviderResult<BucketAggregate>
        {
            Result = countAgg,
        };

Thank!
~john

[jthoni (John T)]

I tried in place creating a new ElasticClient, and another one where I overrode the serializer with JsonNetSerializer.Default. In the debugger, I can see which is using the Nest.InternalSerializer and which is using Nest.JsonNetSerializer. I immediately try serializing with each:

    ElasticClient elasticClient_BuiltInSerializer = new ElasticClient();
                var source = elasticClient_BuiltInSerializer.SourceSerializer.SerializeToString(o);
                var response = elasticClient_BuiltInSerializer.RequestResponseSerializer.SerializeToString(o);

                ConnectionSettings connectionSettings = new ConnectionSettings(new SingleNodeConnectionPool(new Uri("http://fake")), JsonNetSerializer.Default);
                ElasticClient elasticClient_JsonNetSerializer = new ElasticClient(connectionSettings);

                var source2 = elasticClient_JsonNetSerializer.SourceSerializer.SerializeToString(o);
                var response2 = elasticClient_JsonNetSerializer.RequestResponseSerializer.SerializeToString(o);

None of the serialized string values contain "doc_count", "key" or "aggregations."

[jthoni (John T)]

I am betting this is related to this:
[Getting empty objects in json after serialzing DateHistogram Aggregation object from SearchTemplateAsync response · Issue #3093 · elastic/elasticsearch-net · GitHub](Getting empty objects in json after serialzing DateHistogram Aggregation object from SearchTemplateAsync response · Issue #3093 · elastic/elasticsearch-net · GitHub)

[jthoni (John T)]

To read into this further, from the aforementioned github post:

Now, an issue here is that the aggregate types only have read (deserialization) implementations, meaning they have not purposely supported writing (serialization) again to JSON.

The suggested workarounds are to either use the lowlevel client to make the request and then get the response back as a StringResponse, or to DisableDirectStreaming, which would make the request available off of the ISearchResponse. This works fine if I am just wanting to get straight to the json string of the response. That is not the case here.

What I have is one service calling another. The second one makes the query to Elasticsearch and gets the response. I need this to be in a typed Nest object so that we can work with it. We need to be able to traverse the Aggregate buckets, make decisions, then pull out the aggs that fit the criteria, and then return the as a serialized string. The calling app is expecting there to be "keys" and "doc_counts" in the returned object. I can get the string, but I am just going to have to deserialize into an object to work with it, at which point I am in the same place.