Does utf8json used in elastic v7.x support user defined custom formatters?

[stoj (tibor)]

I've got an interesting situation whereby elastic v7 default utf8json has managed to serialize my object, but is unable to de-serialize it correctly.

    public class MyClass : FlagsSet
      {
        [JsonIgnore]
        public bool IsActive
        {
          get
          {
            return this.IsSet("active");
          }
        }
    }

      public class FlagsSet : ICollection<string>, IEnumerable<string>, IEnumerable
      {
        private readonly HashSet<string> _list = new HashSet<string>((IEqualityComparer<string>) StringComparer.InvariantCultureIgnoreCase);
      ...
        public void Add(string item)
        {
          if (string.IsNullOrEmpty(item))
            return;
          this._list.Add(item);
        }
    }

If I was using json.net I'd handle this by writing a converter, but I'm not able to see an equivalent using utf8json as it appears the formatters used by the default serializer (DefaultHighLevelSerializer) are all registered internally. I've read a few pages on customer serializers, most notably these..
https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/custom-serialization.html
https://github.com/elastic/elasticsearch-net/issues/3493

So in short..

1. Is it possible to register a custom utf8json formatter (i.e. similar to the existing json.net converter support via ConnectionSettingsAwareSerializerBase.CreateJsonConverters)? And if so, are you able to point me to an example please?
2. Alternatively, if it's not possible then is there a way to get the utf8json deserialisation to work correctly with the above example?

Thanks in advance

[forloop (Russ Cam)]

The custom serialization documentation linked to has an example of hooking up JsonNetSerializer to use to serialize your documents. This serializer uses JSON.net to serialize documents, so a custom converter can be written and added to it's settings to serialize in a custom fashion.

[stoj (tibor)]

Hi Russ

I'm appreciate ES allows a user defined custom json.net serializer (aka converter), but what I'd like to know is does ES support the equivalent registering of user defined custom UTF8JSON serializers (aka formatters)?

I believe this is a crucial piece of functionality now that ES has changed their default seriazliation library since v7. Especially in light of serialization vs deserialization mismatches (like my earlier example), whereby users (such as myself) need to write custom formatters to workaround it.

If it helps, I believe this is the relevant class/lines of code whereby ES's formatters are defined.. but I'm unable to see how users can define/register their own.

Hopefully it's my just utf8json naivety as I've not used it previously. Although as pointed out the github issue I linked.. it looks like an awesome high performant replacement for json.net

Cheers

[forloop (Russ Cam)]

@stoj

but what I'd like to know is does ES support the equivalent registering of user defined custom UTF8JSON serializers (aka formatters)?

No it does not. Utf8Json used internally is a fork of Utf8Json and all the types are internal. It's used for serializing requests and responses, and also use to serialize documents too, by default. There is no way to register custom formatters for documents however. Some simple customisation is exposed through attributes e.g. StringEnumAttribute to serialize enums as strings, but nothing more complex. If custom serialization is required, I would recommend looking at JsonNetSerializer and custom converters for NEST 7.x and lower.

For NEST 8.x, we are looking at System.Text.Json for serialization and would likely expose custom formatters/converters for this.

[stoj (tibor)]

Thanks for clarifying. Perhaps it might be worth adding a note to the custom serialization help page to help out others?

I look forward to seeing the NEST v8's System.Text.Json. Especially if it brings back support for user defined custom converters.. a must have feature when trying to serialize and deserialize third party models.

[forloop (Russ Cam)]

Custom converters are supported in 7.x using Nest.JsonNetSerializer package and JsonNetSerializer. You could write an IElasticsearchSerializer implementation that uses Utf8Json or System.Text.Json too if preferred