Does utf8json used in elastic v7.x support user defined custom formatters?

No it does not. Utf8Json used internally is a fork of Utf8Json and all the types are internal. It's used for serializing requests and responses, and also use to serialize documents too, by default. There is no way to register custom formatters for documents however. Some simple customisation is exposed through attributes e.g. StringEnumAttribute to serialize enums as strings, but nothing more complex. If custom serialization is required, I would recommend looking at JsonNetSerializer and custom converters for NEST 7.x and lower.

For NEST 8.x, we are looking at System.Text.Json for serialization and would likely expose custom formatters/converters for this.