The pain of serializing unions and closed class hierarchies with System.Text.Json

.NET 11 and C# 15 are adding support for both union types and closed class hierarchies. In this post I look at how System.Text.Json handles these new type hierarches, the differences between them, as well as some of the sharp edges to watch out for. It's probably just me, but I found the whole experience quite painful.

I start by giving a recap on union types and closed class hierarchies. I then create simple examples of each, and try to serialize them with System.Text.Json. I then play whack-a-mole for a while with the errors, and in the end concede that I just can't make it work. Finally, we look at the deserialization path.

This post was written with .NET 11 preview 7. I hope that many of these edge cases are ironed out prior to the final release of .NET 11, but I wouldn't be surprised if more complete support ends up being pushed back to .NET 12, so bear that in mind. And if I'm just missing something obvious, please let me know in the comments!

Union types in C# 15 with the `union` keyword

Unions are one of those basic data structures which are used all the time in the functional programming world; they're available in F#, TypeScript, Rust…pretty much any functional-first language. There are many different types of union, but at their core they allow having a single type that can represent two different things.

For example, imagine we have three different record types, containing different properties, representing Operating Systems:

public record Windows(string Version);
public record Linux(string Distro, string Version);
public record MacOS(string Name, int Version);

Note that these types don't have to have any values in common. Prior to C# 15, there weren't great options for representing something which could be a Windows or Linux or MaxOS object but in C# 15, we can use the union keyword, as shown below:

//     👇 Use `union` as the type
public union SupportedOS(Windows, Linux, MacOS);
//             👆 List the types that are part of the union

You can then create an instance of the SupportedOS type in a couple of ways:

// You can call new and pass in an instance
SupportedOS os = new SupportedOS(new MacOS("Tahoe", 25));

// Or you can use implict conversion (which calls new() behind the scenes)
SupportedOS os = new MacOS("Tahoe", 25);

The canonical way to work with unions is to use a switch expression to pattern match against the type, and because these are exhaustive, you don't need to include the _ => discard case either:

string GetDescription(SupportedOS os) => os switch
{
    Windows windows => $"Windows {windows.Version}",
    Linux linux => $"{linux.Distro} {linux.Version}",
    MacOS macOS => $"MacOS {macOS.Name} ({macOS.Version})",
}; // note: no discard _ required

Unions can be made from custom types, as in the previous example, but you can also form a union for types you don't own, for example:

public union IntOrString(int, string);

To read more about unions, see my earlier post in which I dig into how they're implemented, and how to optimize them. For now, we'll move onto closed class hierarchies.

What is a "closed class hierarchy"?

A closed class hierarchy is a class hierarchy that can only be defined within a single assembly. Attempting to derive from a closed class from a different assembly is a compilation error. This is easiest to see in action.

Imagine you have the following classes, all in the same assembly:

// Create a closed base class
public closed class Pet
{
    public string Name { get; set; }
}

// Each class derives from the closed Pet class
internal class Dog : Pet
{
    public string FavouriteToy { get; set; } 
}

internal sealed class Cat : Pet
{
    public string FavouriteFood { get; set; }
}

The Dog and Cat classes both derive from Pet. This is all C# 101; the closed keyword isn't really changing anything about that.

Note that when you declare a type as closed, it effectively makes it an abstract class too.

The difference is if we create a new assembly, and try to derive from Pet in a different assembly:

// Assembly 2
public class Horse : Pet { }

then this won't compile! Instead you'll get an error like the following:

error CS9382: 'Horse': cannot use a closed type 'Pet' from another assembly as a base type.

This can be very useful for modelling certain domains, and can simplify the logic in your own code significantly. It also means the compiler can apply many of the same "exhaustiveness" guarantees as for unions, (even though they're conceptually quite different—there's no "hierarchy" in unions).

That gives us the background we need, so now we'll look at how this works with System.Text.Json serialization and deserialization.

Serializing unions and closed hierarchies with System.Text.Json

In this section I'm going to simply create some data that contains a mix of unions and closed hierarchies, and then we'll try to serialize that with System.Text.Json, and we'll handle the fallout.

Defining the data structures

For the unions, we'll use the two examples I showed earlier:

// Basic int/string union
public union IntOrString(int, string);

// Simple POCO record types
public record Windows(string Version);
public record Linux(string Distro, string Version);
public record MacOS(string Name, int Version);

// Union of POCOs
public union SupportedOS(Windows, Linux, MacOS);

For the closed class hierarchy, I wanted to use something slightly more complex than the example I gave above, so in this example we have two layers of closed classes:

Pet (closed)
 |___ Cat
 |___ Dog
      |___ Labrador
      |___ Collie

So we have one abstract closed type (Pet), one "base" type Dog, and three "leaf" types (Cat, Labrador, Collie). Also, I made Pet a public type and the rest internal (as one of the uses I see for closed class hierarchies is modelling library APIs similarly to this):

public closed class Pet
{
    public string Name { get; set; }
}

internal class Dog : Pet
{
    public string FavouriteToy { get; set; } 
}

internal sealed class Labrador : Dog
{
    public bool Hungry { get; } = true;
}

internal sealed class Collie : Dog
{
    public bool Nervous { get; } = true;
}

internal sealed class Cat : Pet
{
    public string FavouriteFood { get; set; }
}

Finally, we have the type we're actually going to serialize, which is just one of each of the hierarchies above:

public class Data
{
    public IntOrString Value { get; set; }
    public SupportedOS Os { get; set; }
    public Pet Pet { get; set; }
}

Testing out the default serialization

Now that we have our data structures, we'll create some data, and try to serialize it. The data object below contains 42 in the IntOrString union, Windows in the SupportedOS union, and a Labrador in the Pet closed class hierarchy:

var data = new Data
{
    Value = new IntOrString(42);
    Os = new(new Windows("11")),
    Pet = new Labrador { Name = "Goose", FavouriteToy = "Ducky" }
};

Then we'll do the most basic serialization, using the default serializer settings:

string serialized = JsonSerializer.Serialize(data);
Console.WriteLine(serialized);

Which gives the following (reformatted slightly to make it easier to read)

{
  "Value": 42,
  "Os": {"Version":"11"},
  "Pet": {"Name":"Goose"}
}

This already shows some interesting things:

The IntOrString instance in "Value" was serialized directly as the number 42, and is not a sub-object like you might expect, i.e. it's not serialized as "Value" : { "Value" : 42 }.
The "OS" property is serialized as a sub object, which is probably what you'd expect.
The "Pet" property doesn't include the child properties of Labrador, only the "Name" property that was declared on the Pet class.

The IntOrString serialization behaviour is quite interesting, in that it treats the union type as a "true" first-class primitive in this case, which can be useful if you're interoperating with JSON APIs where a property really could be an int or a string for example.

The Pet serialization is likely not what we want though, so we'll work on fixing that first.

Trying to fix closed class hierarchy serialization

Currently we're losing a lot of information when we serialize our Labrador instance in the "Pet" property, so we'll try fixing that. This is essentially "normal" polymorphism serialization with System.Text.Json, which means you would expect to need to add [JsonDerivedType] and annotate everything correctly, listing all the possible derived types.

`InferClosedTypePolymorphism` is great…sometimes

However, this is a closed hierarchy, so all the possible derived types are known at compile time. So in .NET 11, there's a new option on JsonSerializerOptions (and in RC1, also on the [JsonPolymorphic] attribute) which should build that hierarchy itself, without needing to be explicit, using InferClosedTypePolymorphism! 🎉

var opts = new JsonSerializerOptions
{
    InferClosedTypePolymorphism = true, // Enable inferring derived types for closed classes
};

string serialized = JsonSerializer.Serialize(data, opts); // 👈 Use the options
Console.WriteLine(serialized);

Unfortunately, when we run this code, we get an exception:

System.InvalidOperationException: The inferred derived type 'Dog' is 
less accessible than the polymorphic base type 'Pet'. Inferred derived
 types must be at least as accessible as the base type.

Hmmm. So the fact that I made the derived types less accessible (internal) than the base (public) is apparently not supported. That seems a little surprising to me, especially as this is the sort of pattern I can see being quite useful for closed hierarchies in general. But apparently we have to make everything either public or internal to keep the serializer happy. FINE.

However, after making everything public instead, and running again, we hit another exception:

System.NotSupportedException: Runtime type 'Labrador' is not supported
 by polymorphic type 'Pet'. Path: $.Pet.

O…K… not sure what that one's about, but maybe it's because Dog isn't a closed type? That kind of makes sense, as the feature is inferring about closed types, and Labrador isn't strictly part of that. So let's mark Dog as a closed type too and try again.

And yes, another exception:

System.InvalidOperationException: Specified type 'Dog' is not a supported
derived type for the polymorphic type 'Pet'. Derived types must not be open
generic type definitions, must be assignable to the base type and cannot be
abstract classes or interfaces unless 'JsonUnknownDerivedTypeHandling.
FallBackToNearestAncestor' is specified.

Given this is explicitly saying a derived type cannot be an abstract class (which is how closed types are implemented), that seems to imply that having "two" layers of hierarchy underneath Pet, and having InferClosedTypePolymorphism detect it automatically simply doesn't work.

That seems like a such a big limitation, but I tried a bunch of different permutations to try to find something that works, and I came up blank.

I suspect it's related to the fact that while I sealed the derived Labrador and Collie types in this case, there's nothing that says you have to do that, and then the total hierarchy isn't known, only the direct descendants of a closed type are guaranteed exhaustive (unless you seal the children).

I created an issue to ask about this behaviour, but I don't anticipate that this will be improved before .NET 11 goes GA, but we'll see!

Ultimately, I found only two options:

Only have a single derived type below the closed class.
Manually specify the derived types.

Given the whole purpose of InferClosedTypePolymorphism is to not have to do that second point, it's a bit of a shame, but ultimately that's what I went with. That let me re-instate my internal derived types too. This is the final hierarchy I ended up with:

// I feel like these _should_ be infered by InferClosedTypePolymorphism, but it doesn't work
[JsonDerivedType(typeof(Collie), typeDiscriminator: nameof(Collie))]
[JsonDerivedType(typeof(Labrador), typeDiscriminator: nameof(Labrador))]
[JsonDerivedType(typeof(Cat), typeDiscriminator: nameof(Cat))]
public closed class Pet
{
    public string Name { get; set; }
}

//       👇 Made this closed for completeness
internal closed class Dog : Pet
{
    public string FavouriteToy { get; set; } 
}

// Collie, Labrador, Cat all defined as before

With those attributes in place, we can go back to our basic serialization:

string serialized = JsonSerializer.Serialize(data, opts);
Console.WriteLine(serialized);

And now we have our Labrador type fully serialized, with the $type discriminator included:

{
  "Value":42,
  "Os": {"Version":"11"},
  "Pet": {"$type":"Labrador", "FavouriteToy":"Ducky", "Hungry":true, "Name":"Goose"}
}

It's time to move on to the deserialization side.

Deserializing unions and closed hierarchies with System.Text.Json

Starting from our serialized data above, we'll begin by doing a simple deserialization:

var serialized = """{"Value":42,"Os":{"Version":"11"},"Pet":{"$type":"Labrador","Hungry":true,"FavouriteToy":"Ducky","Name":"Goose"}}""";

var result = JsonSerializer.Deserialize<Data>(serialized);

Oh no, another exception:

System.Text.Json.JsonException: JSON value type 'Object' is ambiguous for
union type 'SupportedOS' because multiple case types can use this value type.
Specify a custom type classifier to support deserialization.
Path: $.Os | LineNumber: 0 | BytePositionInLine: 18.

So what's going on here?

Understanding `union` deserialization limitations

Our SupportedOS union failed to deserialize, although it seems the IntOrString union did deserialize successfully. That's due to fundamental decisions made when first adding support for unions to System.Text.Json where only union shapes that could be unambiguously identified from the type of JSON value support automatic serialization and deserialization.

So the following, for example, could be automatically deserialized, because each type in the union serializes as something different in the JSON:

IntOrString(int, string) which serialize as Number and String respectively.
DoubleOrGuid(double, Guid) which also serialize as Number and String.
BoolOrDictionary(bool, Dictionary) which serialize as Bool and Object.
Many(decimal, DateTime, bool, Dog) which serialize as Number, String, Bool, and Object!

In contrast, the following don't work, because they serialize as the same type of JSON object

Num(int, long)—both cases are Number.
When(DateTime, DateTimeOffset)—both cases are String.
Pet(Cat, Dog)—both cases are Object.

And based on the rules above, all the cases in SupportedOS are serialized as Object:

public union SupportedOS(Windows, Linux, MacOS);

Which explains why the deserialization blew up, while IntOrString did not. Personally, I really don't like the fact that unions don't round-trip by default. I would rather they defaulted to being "ugly" (e.g. using a $type parameter) than the current approach.

There was a lot of discussion about this default behaviour in the API review, with many people arguing the same position as me. Ultimately, there was more concern for "pretty" serialization for the primitives case, for interoperating with TypeScript APIs. I can understand that, but I feel like the default serialization can only support such a tiny subset of those APIs, that it was the wrong default to fall on. I also wish there was a simple way to just opt into the $type discriminator approach, but I can't find it.

Fixing the `union` deserialization with a custom factory

Just in case you skimmed over the previous section, the tl;dr is that unless you have very simple unions, where the types can be simply determined based on the type of the JSON token, you're screwed.

I wish they had shipped an option to just use the "normal" $type discriminator way for unions, but they seem super focused on the typescript/third-party API scenario (and structural deserialization) that they basically punted on the round-trip scenario.

As it is, I literally couldn't work out a way to force a $type discriminator into the serialization/deserialization path 🙁 That could definitely be my inadequacy, so do let me know if there's an easy way. I hoped adding [JsonPolymorphic] would be sufficient, but it's not legal to apply that to a union

Instead, the only hook I could figure out was to implement a custom JsonTypeClassifierFactory which reads the object into memory, and then has you work out which type it represents. Whether that's even possible will depend on the "shape" of your serialized JSON, but as long as there's a "unique" property for each case, then you can do it.

The following shows an example of a classifier for the SupportedOS type, which relies on Distro and Name as being unique on their cases. If not, we look for a Version property, otherwise we fail. It doesn't feel great, but it's an example of the "structural" matching you're forced into by the lack of an explicit discriminator:

public class SupportedOSClassifier : JsonTypeClassifierFactory<SupportedOS>
{
    public override JsonTypeClassifier CreateJsonClassifier(
        JsonTypeClassifierContext context, JsonSerializerOptions options)
    {
        // Utf8JsonReader has the JSON read and cached in memory
        return static (ref Utf8JsonReader reader) =>
        {
            // All of the case types are Objects
            if (reader.TokenType is not JsonTokenType.StartObject)
            {
                return null;
            }

            bool sawVersion = false;

            // Read the properties until we get to the end of the object
            while (reader.Read() && reader.TokenType is not JsonTokenType.EndObject)
            {
                // These two properties are unique for the case types
                if (reader.ValueTextEquals("Distro"u8)) return typeof(Linux);
                if (reader.ValueTextEquals("Name"u8)) return typeof(MacOS);

                // Version is on _all_ the cases, so doesn't discriminate
                if (reader.ValueTextEquals("Version"u8))
                {
                    // everything has a version, so not a discriminator, but required
                    sawVersion = true;
                }

                reader.Read(); // Advance to the value of the property.
                reader.Skip(); // Skip the value.
            }

            // If we had a Version property, then only one case remaining
            // if not, then we failed to match anything, so fail
            return sawVersion ? typeof(Windows) : null;
        };
    }
}

All that remains is to apply that to the union with the new [JsonUnion] attribute:

[JsonUnion(TypeClassifier = typeof(SupportedOsClassifier))]
public union SupportedOS(Windows, Linux, MacOS);

and with that, our deserialization now works as expected🎉

The `JsonUnionTypeStructuralClassifier` is coming in RC1

It's worth noting that there's a generalised version of the the "structural" classifier I wrote above coming in RC1, called JsonUnionTypeStructuralClassifier. It's already merged, but as RC1 isn't out yet, I haven't tried it out.

That said, based on the unit tests, it looks like it will have a bunch of limitations too, for example:

It won't work if any of your union cases are a union themselves.
It won't work if you don't have a unique property (unless marked required, see below)
It won't work if you have mixed POCO and non-POCO objects (e.g. public union ObjectOrDictionaryUnion(Point, Dictionary<string, int>))

It will likely fill the gap left by the default union support, to the point that I suspect most people will end up slapping it on all their unions. The frustrating thing is that API review called out that this was going to have terrible performance, which makes it even more frustrating for me that we can't just have the $type discriminator instead😅

That said, I don't think the JsonUnionTypeStructuralClassifier will work on my SupportedOS union unless we make a couple of tweaks, by marking various properties required:

public record Windows([property: JsonRequired] string Version);
public record Linux([property: JsonRequired] string Distro, string Version);
public record MacOS([property: JsonRequired] string Name, int Version);

By marking these properties as required, I think it means the JsonUnionTypeStructuralClassifier will be able to deserialize each of the cases, but I'll have to give it a try for sure once RC1 is out.

Overall, the good news is that System.Text.Json will support unions and closed hierarchies in .NET 11, even if none of it really works the way I would have liked😅

Summary

In this post I provided a brief introduction to unions and closed hierarchies and provided a simple test scenario for serialization. I then serialized the data with System.Text.Json and ran into issues with the closed hierarchy. I tried to use InferClosedTypePolymorphism but ultimately found that it just didn't work with the hierarchy I was testing with, and had to resort to manual annotation with [JsonDerivedType].

When deserializing I ran into issues with the defaults chosen for union deserialization, namely that it only works out-of-the-box if each of the union case types serializes to a different JSON primitive i.e. Number/String/Object. If you have two objects—e.g. the classic Pet(Cat, Dog)—then the deserialization fails, and you have to write a manual classifier. I showed how to write such a classifier, and there's a general version that will ship in RC1, but overall that feels like a pit many people will fall into.

Andrew Lock | .NET Escapades Andrew Lock