Exploring the code behind IHttpClientFactory in depth

In this post I take a look at the code in the default implementation of IHttpClientFactory in ASP.NET Core—DefaultHttpClientFactory. We'll see how it ensures that HttpClient instances created with the factory prevent socket exhaustion, while also ensuring that DNS changes are respected.

This post assumes you already have a general idea of IHttpClientFactory and what it's used for, so if it's new to you, take a look at Steve Gordon's series on IHttpClientFactory, or see the docs.

The code in this post is based on the HEAD of the master branch of https://github.com/dotnet/runtime at the time of writing, after .NET 5 preview 7. But the code hasn't changed significantly for 2 years, so I don't expect it to change much soon!

I've taken a couple of liberties with the code by removing null checks, in-lining some trivial methods, and removing some code that's tangential to this discussion. For the original code, see GitHub.

A brief overview of IHttpClientFactory

IHttpClientFactory allows you to create HttpClient instances for interacting with HTTP APIs, using best practices to avoid common issues. Before IHttpClientFactory, it was common to fall into one of two traps when creating HttpClient instances:

Create and dispose of new HttpClient instances as required. This can lead to socket exhaustion due to the TIME_WAIT period required after closing a connection.
Create a singleton HttpClient for the lifetime of the application. This can lead to issues when the DNS record for the HTTP API you're using changes—the changes will not be respected by the HttpClient.

IHttpClientFactory was added in .NET Core 2.1, and solves this issue by separating the management of HttpClient, from the the HttpMessageHandler chain that is used to send the message. In reality, it is the lifetime of the HttpClientHandler at the end of the pipeline that is the important thing to manage, as this is the handler that actually makes the connection

HttpClient and HttpClientHandler pipeline

In addition to simply managing the handler lifetimes, IHttpClientFactory also makes it easy to customise the generated HttpClient and message handler pipeline using an IHttpClientBuilder. This allows you to "pre-configure" a named of typed HttpClient that is created using the factory, for example to set the base address or add default headers:

public void ConfigureServices(IServiceCollection services)
{
    services.AddHttpClient("github", c =>
    {
        c.BaseAddress = new Uri("https://api.github.com/");
    })
    .ConfigureHttpClient(c => 
    {
        c.DefaultRequestHeaders.Add("Accept", "application/vnd.github.v3+json");
        c.DefaultRequestHeaders.Add("User-Agent", "HttpClientFactory-Sample");
    });
}

You can add multiple configuration functions at the HttpClient level, but you can also add additional HttpMessageHandlers to the pipeline. Steve shows how you can create your own handlers in in his series. To add message handlers to a named client, use IHttpClientBuilder.AddHttpMessageHandler<>, and register the handler with the DI container:

public void ConfigureServices(IServiceCollection services)
{
    services.AddHttpClient("github", c =>
    {
        c.BaseAddress = new Uri("https://api.github.com/");
    })
    .AddHttpMessageHandler<TimingHandler>() // This handler is on the outside and executes first on the way out and last on the way in.
    .AddHttpMessageHandler<ValidateHeaderHandler>(); // This handler is on the inside, closest to the request.

    // Add the handlers to the service collection
    services.AddTransient<TimingHandler>();
    services.AddTransient<ValidateHeaderHandler>();
}

When you call ConfigureHttpClient() or AddHttpMessageHandler() to configure your HttpClient, you're actually adding configuration messages to a named IOptions instance, HttpClientFactoryOptions. You can read more about named options here, but the details aren't too important for this post.

That handles configuring the IHttpClientFactory. To use the factory, and create an HttpClient, you first obtain an instance of the singleton IHttpClientFactory, and then you call CreateClient(name), providing the name of the client to create.

If you don't provide a name to CreateClient(), the factory will use the default name, "" (the empty string).

public class MyService
{
    // IHttpClientFactory is a singleton, so can be injected everywhere
    private readonly IHttpClientFactory _factory;
    public MyService(IHttpClientFactory factory)
    {
        _factory = factory;
    }

    public async Task DoSomething()
    {
        // Get an instance of the typed client
        HttpClient client = _factory.CreateClient("github");

        // Use the client...
    }
}

The remainder of this post focus on what happens behind the scenes when you call CreateClient().

Creating an HttpClient and HttpMessageHandler

The CreateClient() method, shown below, is how you typically interact with the IHttpClientFactory. I discuss this method below.

// Injected in constructor
private readonly IOptionsMonitor<HttpClientFactoryOptions> _optionsMonitor

public HttpClient CreateClient(string name)
{
    HttpMessageHandler handler = CreateHandler(name);
    var client = new HttpClient(handler, disposeHandler: false);

    HttpClientFactoryOptions options = _optionsMonitor.Get(name);
    for (int i = 0; i < options.HttpClientActions.Count; i++)
    {
        options.HttpClientActions[i](client);
    }

    return client;
}

This is a relatively simple method on the face of it. We start by creating the HttpMessageHandler pipeline by calling CreateHandler(name), and passing in the name of the client to create. We'll look into that method shortly, as it's where most of the magic happens around handler pooling and lifetimes.

Once you have a handler, a new instance of HttpClient is created and passed to the handler. The important thing to note is the disposeHandler: false argument. This ensures that disposing the HttpClient doesn't dispose the handler pipeline, as the IHttpClientFactory will handle that itself.

Finally, the latest HttpClientFactoryOptions for the named client are fetched from the IOptionsMonitor instance. This contains the configuration functions for the HttpClient that were added in Startup.ConfigureServices(), and sets things like the BaseAddress and default headers.

I discussed using IOptionsMonitor in a previous post. It is useful when you want to load named options in a singleton context, where you can't use the simpler IOptionsSnapshot interface. It also has other change-detection capabilities that aren't used in this case.

Finally, the HttpClient is returned to the caller. Let's look at the CreateHandler() method now and see how the HttpMessageHandler pipeline is created. There's quite a few layers to get through, so we'll walk through it step-by-step.

// Created in the constructor
readonly ConcurrentDictionary<string, Lazy<ActiveHandlerTrackingEntry>> _activeHandlers;;

readonly Func<string, Lazy<ActiveHandlerTrackingEntry>> _entryFactory = (name) =>
    {
        return new Lazy<ActiveHandlerTrackingEntry>(() =>
        {
            return CreateHandlerEntry(name);
        }, LazyThreadSafetyMode.ExecutionAndPublication);
    };

public HttpMessageHandler CreateHandler(string name)
{
    ActiveHandlerTrackingEntry entry = _activeHandlers.GetOrAdd(name, _entryFactory).Value;

    entry.StartExpiryTimer(_expiryCallback);

    return entry.Handler;
}

The CreateHandler() method does two things:

It gets or creates an ActiveHandlerTrackingEntry
It stars a timer on the entry

The _activeHandlers field is a ConcurrentDictionary<>, keyed on the name of the client (e.g. "gitHub"). The dictionary values are Lazy<ActiveHandlerTrackingEntry>. Using Lazy<> here is a neat trick I've blogged about previously to make the GetOrAdd function thread safe. The job of actually creating the handler occurs in CreateHandlerEntry (which we'll see shortly) which creates an ActiveHandlerTrackingEntry.

The ActiveHandlerTrackingEntry is mostly an immutable object containing an HttpMessageHandler and a DI IServiceScope. In reality it also contains an internal timer that is used with the StartExpiryTimer() method to call the provided callback when the timer of length Lifetime expires.

internal class ActiveHandlerTrackingEntry
{
    public LifetimeTrackingHttpMessageHandler Handler { get; private set; }
    public TimeSpan Lifetime { get; }
    public string Name { get; }
    public IServiceScope Scope { get; }
    public void StartExpiryTimer(TimerCallback callback)
    {
        // Starts the internal timer
        // Executes the callback after Lifetime has expired.
        // If the timer has already started, is noop
    }
}

So the CreateHandler method either creates a new ActiveHandlerTrackingEntry, or retrieves the entry from the dictionary, and starts the timer. In the next section we'll look at how the CreateHandlerEntry() method creates the ActiveHandlerTrackingEntry instances:

Creating and tracking HttpMessageHandlers in CreateHandlerEntry

The CreateHandlerEntry method is where the HttpClient handler pipelines are created. It's a somewhat complex method, so I'll show it first, and then talk through it afterwards. The version shown below is somewhat simplified compared to the real version, but it maintains all the salient points.

// The root service provider, injected into the constructor using DI
private readonly IServiceProvider _services;

// A collection of IHttpMessageHandler "configurers" that are added to every handler pipeline
private readonly IHttpMessageHandlerBuilderFilter[] _filters;

private ActiveHandlerTrackingEntry CreateHandlerEntry(string name)
{
    IServiceScope scope = _services.CreateScope(); 
    IServiceProvider services = scope.ServiceProvider;
    HttpClientFactoryOptions options = _optionsMonitor.Get(name);

    HttpMessageHandlerBuilder builder = services.GetRequiredService<HttpMessageHandlerBuilder>();
    builder.Name = name;

    // This is similar to the initialization pattern in:
    // https://github.com/aspnet/Hosting/blob/e892ed8bbdcd25a0dafc1850033398dc57f65fe1/src/Microsoft.AspNetCore.Hosting/Internal/WebHost.cs#L188
    Action<HttpMessageHandlerBuilder> configure = Configure;
    for (int i = _filters.Length - 1; i >= 0; i--)
    {
        configure = _filters[i].Configure(configure);
    }

    configure(builder);

    // Wrap the handler so we can ensure the inner handler outlives the outer handler.
    var handler = new LifetimeTrackingHttpMessageHandler(builder.Build());

    return new ActiveHandlerTrackingEntry(name, handler, scope, options.HandlerLifetime);

    void Configure(HttpMessageHandlerBuilder b)
    {
        for (int i = 0; i < options.HttpMessageHandlerBuilderActions.Count; i++)
        {
            options.HttpMessageHandlerBuilderActions[i](b);
        }
    }
}

There's quite a lot to unpack here. The method starts by creating a new IServiceScope using the root DI container. This creates a DI scope, so that scoped services can be sourced from the associated IServiceProvider. We also retrieve the HttpClientFactoryOptions for the requested HttpClient name, which contains the specific handler configuration for this instance.

The next item retrieved from the container is an HttpMessageHandlerBuilder, which by default is a DefaultHttpMessageHandlerBuilder. This is used to build the handler pipeline, by creating a "primary" handler, which is the HttpClientHandler that is responsible for making the socket connection and sending the request. You can add additional DelegatingHandlers that wrap the primary server, creating a pipeline for requests and responses.

Image of the delegating handlers and primary handler

The DelegatingHandlers are added to the builder using a slightly complicated arrangement, that is very reminiscent of how the middleware pipeline in an ASP.NET Core app is built:

The Configure() local method builds a pipeline of DelegatingHandlers, based on the configuration you provided in Startup.ConfigureServices().
IHttpMessageHandlerBuilderFilter are filters that are injected into the IHttpClientFactory constructor. They are used to add additional handlers into the DelegatingHandler pipeline.

IHttpMessageHandlerBuilderFilter are directly analogous to the IStartupFilters used by the ASP.NET Core middleware pipeline, which I've talked about previously. A single IHttpMessageHandlerBuilderFilter is registered by default, the LoggingHttpMessageHandlerBuilderFilter. This filter adds two additional handlers to the DelegatingHandler pipeline:

LoggingScopeHttpMessageHandler at the start (outside) of the pipeline, which starts a new logging scope.
A LoggingHttpMessageHandler at the end (inside) of the pipeline, just before the request is sent to the primary HttpClientHandler, which records logs about the request and response.

Finally, the whole pipeline is wrapped in a LifetimeTrackingHttpMessageHandler, which is a "noop" DelegatingHandler. We'll come back to the purpose of this handler later.

The code (and probably my description) for CreateHandlerEntry is definitely somewhat hard to follow, but the end result for an HttpClient configured with two custom handlers (as demonstrated at the start of this post) is a handler pipeline that looks something like the following:

The final build handler pipeline

Once the handler pipeline is complete, it is saved in a new ActiveHandlerTrackingEntry instance, along with the DI IServiceScope used to create it, and given the lifetime defined in HttpClientFactoryOptions (two minutes by default).

This entry is returned to the caller (the CreateHandler() method), added to the ConcurrentDictionary<> of handlers, added to a new HttpClient instance (in the CreateClient() method), and returned to the original caller.

For the next two minutes, every time you call CreateClient(), you will get a new instance of HttpClient, but which has the same handler pipeline as was originally created.

Each named or typed client gets its own message handler pipeline. i.e. two instances of the "github" named client will have the same handler chain, but the "api" named client would have a different handler chain.

The next challenge is cleaning up and disposing the handler chain once the two minute timer has expired. This requires some careful handling, as you'll see in the next section.

Cleaning up expired handlers.

After two minutes, the timer stored in the ActiveHandlerTrackingEntry entry will expire, and fire the callback method passed into StartExpiryTimer(): ExpiryTimer_Tick().

ExpiryTimer_Tick is responsible for removing the active handler entry from the ConcurrentDictionary<> pool, and adding it to a queue of expired handlers:

// Created in the constructor
readonly ConcurrentQueue<ExpiredHandlerTrackingEntry> _expiredHandlers;

// The Timer instance in ActiveHandlerTrackingEntry calls this when it expires
internal void ExpiryTimer_Tick(object state)
{
    var active = (ActiveHandlerTrackingEntry)state;

     _activeHandlers.TryRemove(active.Name, out Lazy<ActiveHandlerTrackingEntry> found);

    var expired = new ExpiredHandlerTrackingEntry(active);
    _expiredHandlers.Enqueue(expired);

    StartCleanupTimer();
}

Once the active handler is removed from the _activeHandlers collection, it will no longer be handed out with new HttpClients when your call CreateClient(). But there are potentially HttpClients out there which are still using the active handler. The IHttpClientFactory has to wait for all the HttpClient instances that reference this handler to be cleaned up before it can dispose the handler pipeline.

The problem is, how can IHttpClientFactory track that the handler is no longer referenced? The key is the use of the "noop" LifetimeTrackingHttpMessageHandler, and the ExpiredHandlerTrackingEntry.

The ExpiredHandlerTrackingEntry, shown below, creates a WeakReference to the LifetimeTrackingHttpMessageHandler. As I showed in the previous section, the LifetimeTrackingHttpMessageHandler is the "outermost" handler in the chain, so it is the handler that is directly referenced by the HttpClient.

internal class ExpiredHandlerTrackingEntry
{
    private readonly WeakReference _livenessTracker;

    // IMPORTANT: don't cache a reference to `other` or `other.Handler` here.
    // We need to allow it to be GC'ed.
    public ExpiredHandlerTrackingEntry(ActiveHandlerTrackingEntry other)
    {
        Name = other.Name;
        Scope = other.Scope;

        _livenessTracker = new WeakReference(other.Handler);
        InnerHandler = other.Handler.InnerHandler;
    }

    public bool CanDispose => !_livenessTracker.IsAlive;

    public HttpMessageHandler InnerHandler { get; }
    public string Name { get; }
    public IServiceScope Scope { get; }
}

Using WeakReference to the LifetimeTrackingHttpMessageHandler means that the only direct references to the outermost handler in the chain are in HttpClients. Once all these HttpClients have been collected by the garbage collector, the LifetimeTrackingHttpMessageHandler will have no references, and so will also be disposed. The ExpiredHandlerTrackingEntry can detect that via the WeakReference.IsAlive property.

Note that the ExpiredHandlerTrackingEntry does maintain a reference to the rest of the handler pipeline, so that it can properly dispose the inner handler chain, as well as the DI IServiceScope.

After an entry is added to the _expiredHandlers queue, a timer is started by StartCleanupTimer() which fires after 10 seconds. This calls the CleanupTimer_Tick() method, which checks to see whether all the references to the handler have expired. If so, the handler chain and IServiceScope are disposed. If not, they are added back onto the queue, and the clean-up timer is started again:

internal void CleanupTimer_Tick()
{
    // Stop any pending timers, we'll restart the timer if there's anything left to process after cleanup.
    StopCleanupTimer();

    // Loop through all the timers
    int initialCount = _expiredHandlers.Count;
    for (int i = 0; i < initialCount; i++)
    {
        _expiredHandlers.TryDequeue(out ExpiredHandlerTrackingEntry entry);

        if (entry.CanDispose)
        {
            // All references to the handler chain have been removed
            try
            {
                entry.InnerHandler.Dispose();
                entry.Scope?.Dispose();
            }
            catch (Exception ex)
            {
                // log the exception
            }
        }
        else
        {
            // If the entry is still live, put it back in the queue so we can process it
            // during the next cleanup cycle.
            _expiredHandlers.Enqueue(entry);
        }
    }

    // We didn't totally empty the cleanup queue, try again later.
    if (_expiredHandlers.Count > 0)
    {
        StartCleanupTimer();
    }
}

The method presented above is pretty simple - it loops through each of the ExpiredHandlerTrackingEntry entries in the queue, and checks if all references to the LifetimeTrackingHttpMessageHandler handlers have been removed. If they have, the handler chain (everything that was inside the lifetime tracking handler) is disposed, as is the DI IServiceScope.

If there are still live references to any of the LifetimeTrackingHttpMessageHandler handlers, the entries are put back in the queue, and the cleanup timer is started again. Every 10 seconds another cleanup sweep is run.

I simplified the CleanupTimer_Tick() method shown above compared to the original. The original adds additional logging, and uses locking to ensure only a single thread runs the cleanup at a time.

That brings us to the end of this deep dive in the internals of IHttpClientFactory! IHttpClientFactory shows the correct way to manage HttpClient and HttpMessageHandlers in your application. Having read through the code, it's understandable that noone got this completely right for so long - there's a lot of tricky gotchas there! If you've made it this far, I suggest going and looking through the original code, I highlighted (what I consider) the most important points in this post, but you can learn a lot from reading other people's code!

Summary

In this post I looked at the source code behind the default IHttpClientFactory implementation in .NET Core 3.1, DefaultHttpClientFactory. I showed how the factory stores an active HttpMessageHandler pipeline for each configured named or typed client, with each new HttpClient getting a reference to the same pipeline. I showed that the handler pipeline is built in a similar way to the ASP.NET Core middleware pipeline, using handlers. Finally, I showed how the factory tracks whether any HttpClients reference a pipeline instance by using a WeakReference to the handler.

Andrew Lock | .NET Escapades Andrew Lock