Saving source generator output in source control

In this post I describe how to persist the output of your source generator to disk so that it can be part of source control and code reviews, how to control where the files are output, and how to handle the case where your source generator produces different output depending on the target framework.

Source generators don't produce artifacts by default

One of the big selling points about source-generators is that they run in the compiler. That makes them more convenient than other source generation techniques, such as t4 templates, as you don't need a separate build step.

However, one potential disadvantage also stems from the fact the source generator runs inside the compiler. That can make it hard to see the effect of a source generator when you're not in the context of an IDE.

For example, if you're reviewing a pull request on GitHub that uses source generators, and you make a change that adds code to the project, you may find it useful to have that output visible in the PR. This may be especially important for "critical" code.

For example, in the Datadog Tracer we recently started using source generators to generate methods called by the "native" part of the profiler, that controls which integrations are enabled. This is a crucial part of the tracer so it's important to see any changes. We wanted any changes to be visible in PRs, so we needed to make sure the source generator output was written to files.

Emitting compiler generated files

There's a simple switch to enable persisting source generator files to the file system: EmitCompilerGeneratedFiles. You can set this property in your project file:

<PropertyGroup>
    <EmitCompilerGeneratedFiles>true</EmitCompilerGeneratedFiles>
</PropertyGroup>

Or you can set the MSBuild property in any other way, e.g. at the command line when building

dotnet build /p:EmitCompilerGeneratedFiles=true

When you set this property alone, the compiler will output the hint files to disk. For example, if we consider the NetEscapades.EnumGenerators package, and enable the EmitCompilerGeneratedFiles property, we can see that the source generated files are written to the obj folder:

Generated files in the obj folder

Specifically, the source generator output is written to a folder defined as:

{BaseIntermediateOutpath}/generated/{Assembly}/{SourceGeneratorName}/{GeneratedFile}

In the example above, we have

BaseIntermediateOutpath: obj/Debug/net6.0
Assembly: NetEscapades.EnumGenerators
SourceGeneratorName: NetEscapades.EnumGenerators.EnumGenerator
GeneratedFile: ColoursExtensions_EnumExtensions.g.cs, EnumExtensionsAttribute.g.cs

Writing files to the obj folder is all well and good, but it doesn't really solve our problem, as the bin and obj folders are typically excluded from source control. We could explicitly include them into source control, but a better option is to emit the files somewhere else.

Controlling the output location

You can control the location of the compiler emitted files by setting the CompilerGeneratedFilesOutputPath property. This is a path relative to the project root folder. So for example, if you set the following in your project file:

<PropertyGroup>
    <EmitCompilerGeneratedFiles>true</EmitCompilerGeneratedFiles>
    <CompilerGeneratedFilesOutputPath>Generated</CompilerGeneratedFilesOutputPath>
</PropertyGroup>

This will write the files to the Generated folder in the project folder:

Generated files in the 'Generated' folder

Whatever you place in CompilerGeneratedFilesOutputPath replaces the {BaseIntermediateOutpath}/generated prefix in the file path, so the files are written to:

{CompilerGeneratedFilesOutputPath}/{Assembly}/{SourceGeneratorName}/{GeneratedFile}

On the face of it, this seems like it solves all the issues: the source generator contents are emitted to the file system, to a place that's included in source control. Problem solved right?

The difficulty is when you try and build for a second time, after the files have already been written, you'll get a number of errors:

ColoursExtensions_EnumExtensions.g.cs(31,28): error CS0111: Type 'ColoursExtensions' already defines a member called 'IsDefined' with the same parameter types ColoursExtensions_EnumExtensions.g.cs(40,28): error CS0111: Type 'ColoursExtensions' already defines a member called 'TryParse' with the same parameter types

That's because the compiler is including the emitted files in addition to the in-memory source generator output. This causes duplication of the types and the errors above. The answer is to exclude the files from the compilation.

Excluding emitted files from the compilation

The simple solution to this problem is to remove the emitted files from the project compilation, so that only the in-memory source generator output is part of the compilation. You can exclude these individually (e.g. by right-clicking the file in Visual Studio), or more usefully, you can use a wildcard pattern to exclude all the .cs files in those folders:

<PropertyGroup>
    <EmitCompilerGeneratedFiles>true</EmitCompilerGeneratedFiles>
    <CompilerGeneratedFilesOutputPath>Generated</CompilerGeneratedFilesOutputPath>
</PropertyGroup>

<ItemGroup>
    <!-- Exclude the output of source generators from the compilation -->
    <Compile Remove="$(CompilerGeneratedFilesOutputPath)/**/*.cs" />
</ItemGroup>

With this change, we now have the best of all worlds—the source generator output is emitted to disk, it is included in source control so can be reviewed in PRs etc, and it doesn't impact the compilation itself.

Splitting by target framework

The properties above are what we initially used when adding our first source generator in the Datadog Tracer. However, this subsequently caused us a bit of an issue.

For context, the Datadog Tracer currently supports multiple target frameworks: net461, netstandard2.0, netcoreapp3.1. However some of our integrations are only applicable for specific target frameworks. For example, the ASP.NET integration only applies to net461, so we use #if NETFRAMEWORK to exclude it from the .NET Core assembly.

The difficulty is that the output of our source generator is different for each target framework, yet the output of each target framework compilation is written into the same folder in all cases. Each time the compiler runs for a target framework, it overwrites the existing file output in Generated/AssemblyName/GeneratorName/FileName.cs! Three different outputs of the source generator, but only one of those is persisted to disk.

To work around this problem, we added the target framework to the output file path using the $(TargetFramework) property.

<PropertyGroup>
    <!-- Persist the source generator (and other) files to disk -->
    <EmitCompilerGeneratedFiles>true</EmitCompilerGeneratedFiles>
    <!-- 👇 The "base" path for the source generators -->
    <GeneratedFolder>Generated</GeneratedFolder>
    <!-- 👇 Write the output for each target framework to a different sub-folder -->
    <CompilerGeneratedFilesOutputPath>$(GeneratedFolder)\$(TargetFramework)</CompilerGeneratedFilesOutputPath>
</PropertyGroup>

<ItemGroup>
    <!-- 👇 Exclude everything in the base folder -->
    <Compile Remove="$(GeneratedFolder)/**/*.cs" />
</ItemGroup>

With this change, the output of the source generator for each framework is written into a separate folder, so we can easily see the difference between the assemblies.

Splitting files by target framework

Obviously this approach isn't necessary unless you're multi-targeting and you produce different source-generator output for different target frameworks, but it's an easy approach if you are.

Summary

In this post I described how you can ensure source generators emit their generated outputs to disk. This can be useful if you want to monitor for changes in the source generator output, or want to be able to review that output in a non-IDE scenario, such as in a pull request on GitHub. I then showed how to control where the files are written, and one approach to handle the case where the source generator creates different output for different target framework builds of your project.

Andrew Lock | .NET Escapades Andrew Lock