In this post I provide an introduction to the System.Diagnostics.Metrics API, show how to use dotnet-counters for local monitoring of metrics, and show how to add a custom metric to your application.
The System.Diagnostics.Metrics APIs
The System.Diagnostics.Metrics APIs were originally introduced as a built-in in feature in .NET 6, but are also supported in earlier versions of .NET Core and .NET Framework using the System.Diagnostics.DiagnosticSource NuGet package. The metrics APIs provide a way to both create and report on metrics generated by an application, such as simple counters, gauges, or histograms of values. I'll describe each of the available metric types later.
The System.Diagnostics.Metrics APIs are designed to easily interoperate with OpenTelemetry, and so can be consumed by a large range of applications. You can also read the metrics using .NET SDK tools like dotnet-counters.
The word "metric" is often used in multiple different ways. Is it a single "point" with associated "tags"? Is it the full set of the recorded values for a single concept? Is it the "aggregated" statistics for all of these points? It's common to see both meanings. In this post I mainly use "metric" to mean a stream of recordings for a single concept.
There are two core concepts exposed by the System.Diagnostics.Metrics APIs. These are Instruments and Meters:
Instrument: An instrument records the values for a single metric of interest. You might have separateInstruments for "products sold", "invoices created", "invoice total", and "GC heap size".Meter: AMeteris a logical grouping of multiple instruments. For example, theSystem.RuntimeMetercontains multipleInstruments about the workings of the runtime, while theMicrosoft.AspNetCore.HostingMetercontainsInstruments about the HTTP requests received by ASP.NET Core.
There are also several different types of Instrument:
Counter<T>/ObservableCounter<T>: These represent a count of occurrences, so return a non-negative values. For example, the number of requests received might be aCounter<T>.UpDownCounter<T>/ObservableUpDownCounter<T>: These are similar to a counter, but can be used to record both positive and negative values. This may be used to report the change in queue size or the number of active requests.Gauge<T>/ObservableGauge<T>: These return a value that represents the "current value". The values it emits effectively "replace" the previous value. For example, the amount of memory used might be aGauge<T>.Histogram<T>: Reports arbitrary values, which could be subsequently processed to calculate further statistics, or plot as a graph. For example, the duration of each request might be recorded as aHistogram.
You'll note that the Counter<T>, UpDownCounter<T>, and Gauge<T> all have observable versions. This difference relates to how the Instrument records and emits values; observable instruments only retrieve their values when explicitly requested, whereas the non-observable versions emit a value as soon as that value is recorded.
The choice of whether an
Instrumentshould be implemented asObservable*is driven partly by performance considerations, and partly by how the value is obtained. I'll cover more about the implementation differences with observableInstruments in a future post.
Collecting metrics with dotnet-counters
When running in production, you'll likely want to collect your metrics using an OpenTelemetry exporter integration or another solution (e.g. Datadog can collect these metrics without requiring application changes), but for local testing dotnet-counters is a very convenient tool.
dotnet-counters is a .NET tool shipped by Microsoft that you can install by running:
dotnet tool install -g dotnet-counters
You can then run the tool by specifying a process ID or process name to monitor, using:
dotnet-counters monitor -n MyApp
# or
dotnet-counters monitor -p 123
Alternatively, you can specify a command to run when starting the tool, and it will monitor the target process:
dotnet-counters monitor -- dotnet MyApp.dll
When you run dotnet-counters in this "monitor" mode, the counter values are written to the console and periodically refresh:
Press p to pause, r to resume, q to quit.
Status: Running
Name Current Value
[System.Runtime]
dotnet.assembly.count ({assembly}) 100
dotnet.gc.collections ({collection})
gc.heap.generation
------------------
gen0 67
gen1 6
gen2 1
dotnet.gc.heap.total_allocated (By) 4,134,656
dotnet.gc.last_collection.heap.fragmentation.size (By)
gc.heap.generation
------------------
gen0 911,896
gen1 5,544
gen2 1,656
loh 0
poh 0
dotnet.gc.last_collection.heap.size (By)
gc.heap.generation
------------------
gen0 943,560
gen1 271,288
gen2 840,136
loh 0
poh 24,528
dotnet.gc.last_collection.memory.committed_size (By) 3,981,312
dotnet.gc.pause.time (s) 0.106
dotnet.jit.compilation.time (s) 1.096
dotnet.jit.compiled_il.size (By) 199,280
dotnet.jit.compiled_methods ({method}) 2,126
dotnet.monitor.lock_contentions ({contention}) 1
dotnet.process.cpu.count ({cpu}) 4
dotnet.process.cpu.time (s)
cpu.mode
--------
system 5.453
user 9.313
dotnet.process.memory.working_set (By) 51,384,320
dotnet.thread_pool.queue.length ({work_item}) 0
dotnet.thread_pool.thread.count ({thread}) 4
dotnet.thread_pool.work_item.count ({work_item}) 61,911
dotnet.timer.count ({timer}) 0
You can choose which Meters to display by passing a comma-separated list of counters using --counters, for example to show the Microsoft.AspNetCore.Hosting Meter, you would use:
> dotnet-counters monitor --counters 'Microsoft.AspNetCore.Hosting' -- dotnet MyApp.dll
Press p to pause, r to resume, q to quit.
Status: Running
Name Current Value
[Microsoft.AspNetCore.Hosting]
http.server.active_requests ({request})
http.request.method url.scheme
------------------- ----------
GET http 0
http.server.request.duration (s)
http.request.method http.response.status_code http.route network.protocol.version url.scheme Percentile
------------------- ------------------------- ---------- ------------------------ ---------- ----------
GET 200 / 1.1 http 50 0
GET 200 / 1.1 http 95 0
GET 200 / 1.1 http 99 0
The metrics in the above image were created by hitting the same endpoint in a sample app several times, but they show some of the different features of the Instruments available. Each metric has an associated unit ({request} and s), and also an associated set of tags. Tags are an important aspect when recording metrics, as they allow you to more easily group and segregate data.
For example, the http.server.active_requests up/down counter has tags for http.request.method and url.scheme. Seeing as I only made GET requests to http://localhost:5000, you only see one set of tags. But if I had made POST requests, or requests using https then you would have seen other values there. Similarly, the values in the http.server.request.duration histogram include tags for each value.
Managing tag cardinality (the number of possible values) is an important aspect of dealing with tags in all observability data. Depending on how your data is stored, large tag cardinality could cause large data storage costs and an impact on performance. Those limits will generally be controlled by whatever system you're exporting your metrics to.
As well as "immediate" monitoring approaches like the example above, which just outputs to the console, dotnet-counters also has options for just collecting the metrics and exporting them in a variety of formats. You could drive a production monitoring system this way, but I suspect most usages of dotnet-counters are for the local testing scenario.
Creating your own metrics
The dotnet-counters example above demonstrates some of the built-in metrics available in .NET 10. The System.Runtime meter is available since .NET 9, and the Microsoft.AspNetCore.Routing meter is available since .NET 8, but there are many other additional built-in metrics available in different versions of .NET. You can find what's available here:
These metrics can provide a reasonable overview of how your system is operating in general, but there might also be application-specific or business related metrics that would be useful to record from the application itself.
As an example, we'll create a very simple counter metric that just records the number of requests sent to a particular API. To make it slightly less abstract, we'll imagine this to be a product pricing endpoint, and we want to track how often the details are checked for a given product.
Creating the initial app
We'll start by creating the basic app using
dotnet new web
and updating the application to the following:
var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();
app.MapGet("/product/{id}", (int id) =>
{
// This would return the real details
// TODO: add metrics
return $"Pricing for product {id}";
});
app.Run();
We would obviously return real pricing details from this API, but this is just a demo after all.
Creating our Instrument and Meter
Now let's add our metrics. We need to create two things:
- An
Instrumentto track the number of requests. - A
Meterto hold our instrument (and any future related instruments).
We need to be careful about the naming of both of these, as they essentially serve as the public API for subsequent consumers of our metrics.
Seeing as this is an ASP.NET Core application and we generally avoid global static variables, the example below shows how we would create a class to encapsulate our Instrument and Meter, so that we can register it with the dependency injection container later. If you were creating an app that doesn't use DI, you could just as easily use new Meter(), and save the variable in a global variable.
public class ProductMetrics
{
private readonly Counter<long> _pricingDetailsViewed;
public ProductMetrics(IMeterFactory meterFactory)
{
var meter = meterFactory.Create("MyApp.Products");
_pricingDetailsViewed = meter.CreateCounter<long>("myapp.products.pricing_page_requests");
}
public void PricingPageViewed(int id)
{
_pricingDetailsViewed.Add(delta: 1, new KeyValuePair<string, object?>("product_id", id));
}
}
In the code above we:
- Create a new
MetercalledMyApp.Products. This is named following similar guidelines to the built-in meters; we have "namespaced" using our app's name, and the broad category of the instruments it will include. - We create a
Counter<long>calledmyapp.products.pricing_page_requests. This is named using the OpenTelemetry naming guidelines. I opted forlongbecause I anticipate that some pages will get a lot of reviews in the lifetime of the app (more thanint.MaxValue). - We added a convenience method for recording a view of a product's pricing page, tagging the view with the ID of the product we're viewing. We could add other tags&mash;maybe the product name would be more useful for example&smash;but this tag will do for our purposes.
If we want to add additional Instruments to the same Meter later, we would create them here, and likely add similar convenience methods.
Hooking up the Instrument in the app
Now that we have our metrics helper, we need to make use of it in our app. This involves both registering the helper in DI, and using it in our API:
using System.Diagnostics.Metrics;
using Microsoft.Extensions.Diagnostics.Metrics;
var builder = WebApplication.CreateBuilder(args);
// 👇 Register in DI
builder.Services.AddSingleton<ProductMetrics>();
var app = builder.Build();
// Inject in API handler 👇
app.MapGet("/product/{id}", (int id, ProductMetrics metrics) =>
{
metrics.PricingPageViewed(id); // 👈 Record
return $"Details for product {id}";
});
app.Run();
We can now try it out using dotnet-counters to view the metrics.
Testing our new metric
We'll start by running our app:
dotnet run
and then in a separate terminal window, we'll set dotnet-counters running using
dotnet-counters monitor -n MyApp --counters MyApp.Products
I've used the -n option to find the app by name, MyApp, and made sure to only show the MyApp.Products instrument.
If we hit the product endpoint a few times with various IDs, we can see that the metrics are reported to dotnet-counters as expected!
With that, we have confirmed that we have a custom metric being successfully recorded 🎉
Adding extra information for consumers
In the dotnet-counters output above, we can see that the Instrument is reported with the unit Count, inferred from the instrument type. That's fine, but the Instrument API lets us provide additional details that can be optionally used by consumers to customise the display or metrics.
For example, we could add some additional details to our instrument, as follows:
_pricingDetailsViewed = meter.CreateCounter<int>(
"myapp.products.pricing_page_requests",
unit: "requests",
description: "The number of requests to the pricing details page for the product with the given product_id");
If we run and monitor the app again, the dotnet-counters output has changed slightly. The unit for myapp.products.pricing_page_requests has changed to requests instead of Count:
Name Current Value
[MyApp.Products]
myapp.products.pricing_page_requests (requests)
product_id
----------
1 1
234 1
5 4
That's a small nicity, and the description isn't used anywhere by dotnet-counters, but other exporters might choose to use it. Depending on how you're exporting your metrics out of process, your metric should now be available everywhere!
Summary
In this post, I provided an introduction to the System.Diagnostics.Metrics APIs. I described some of the terminology used, such as Meter and Instrument, and the various different types of Instrument available. I then showed how you can use dotnet-counters to monitor the metrics produced by your app, primarily for local investigation. Finally, I showed how you could create a custom metric, customize it, hook it up to dependency injection, and report it in dotnet-counters.
