In this post I'll discuss a technique I use occasionally to ensure that an ASP.NET Core app is able to start correctly, as part of a continuous integration (CI) build pipeline. These "smoke tests" provide an initial indication that there may be something wrong with the build, which may not be caught by other tests.

How useful you find these test will vary depending on your individual app, your integration tests, your build pipeline, and your deployment process. This post is not meant to be prescriptive, it just describes one tool I use in some situations to prevent bad builds being published. Functional/integration tests are another, more thorough approach.

Building and testing ASP.NET Core apps in Docker

One of the selling points of ASP.NET core is the ability to build and run your applications on virtually any operating system. In my case, I typically build ASP.NET Core applications on Windows, and publish them as Docker containers to a private Docker registry as part of the CI build process. Once the application has been automatically tested, it is deployed to a Kubernetes cluster by the CI build system.

Different testing strategies

I've previously written several posts about building ASP.NET Core apps in Docker. I typically use Cake to build and publish my apps as described in this post. A key part of the build process is running unit tests, to verify that your app is working correctly. If any of the unit tests fail, the whole build process should automatically fail, and the deployment of your app should halt.

As well as unit tests, you can also write functional/integration tests for your application. ASP.NET Core 2.1 included the Microsoft.AspNetCore.Mvc.Testing NuGet package to help with functional testing of MVC apps, as described in the documentation. This package simplifies many aspects of functional testing that were non-obvious, and problematic (especially related to Razor compilation).

Functional tests are a great way to test whole "slices" of your application. Unlike unit tests, in which you're often testing a single class or method in isolation, functional tests focus on testing multiple components in combination. For example, you could use functional tests to confirm that when a protected Web API controller is invoked via an HTTP request, unauthenticated users receive a 401 Unauthorized response, while authenticated (and authorized) users receive a 200 Ok response.

Diagram of functional tests

Unfortunately, depending on the design of your application, functional tests could prove difficult to write. I find one of the biggest challenges is when an app requires a database. Depending on which libraries you are using in your app, providing a stub/mocked database can prove a challenge. EF Core has good support for using in-memory databases, which are perfect for this situation. If you're using Dapper however, the solution can be less clear, with no easy way (that I'm aware of!) to replace the underlying database layer.

Just yesterday Jeremy D Miller posted an article on one approach to creating integration tests that use a database, by spinning up database docker containers from inside your .NET code.

Smoke tests for testing app startup

Given that integration testing is not always simple, I wanted a simple way to test that my apps would at least start. Unit tests are great, but pretty much by definition will not catch errors related to how your app is composed. For example, unit tests won't typically be able to tell if you have dependency injection configuration errors in your app.

The solution I settled on involves running the application in a docker container, as part of the CI build process, and calling a basic status/health check endpoint. The act of starting the application will potentially catch any basic configuration issues, while not requiring the presence of infrastructure elements (like the database).

Obviously, by design, this is not a complete test of the app. I'm trying to confirm that I've not made any mistakes that prevent the app running. If you are already running integration tests, then adding this smoke test probably isn't going to tell you a lot, but then it won't (shouldn't?) hurt!

Docker and Kubernetes both include the concept of a readiness/liveness probe for applications. The smoke-test approach essentially runs the readiness probe as part of the build process, instead of waiting for deploy time.

At this point I'll assume you already have the following configured:

  1. An ASP.NET Core app, running in Docker
  2. A CI build pipeline, which produces runnable Docker images of your app. The pipeline must also be capable of running Docker containers
  3. A status/health check endpoint in your app that does not require additional infrastructure (i.e. a database)

The status/health check endpoint could be a simple piece of middleware, an MVC controller, or the health check service/middleware in the ASP.NET Core 2.2 preview.

Once you have all these in place, adding the smoke test as part of your CI build pipeline is relatively simple. In the next section I'll describe the script I use, and how to call it.

The smoke test Bash script

The complete Bash script for running smoke tests is shown below. I'll walk through the script in detail afterwards, as it's quite verbose, but a large amount of the script is boilerplate for setting variables and capturing errors etc, so try not to feel too overwhelmed!

#!/bin/bash

# treat undefined variables as an error, and exit script immediately on error
set -eux

APP_NAME="$1"
APP_IMAGE="$2"
TEST_PATH="$3"
SMOKE_TEST_IMAGE="$APP_NAME-smoke-test"

TEST_PORT=80
MAX_WAIT_SECONDS=30

# kill any lingering test containers
docker kill $SMOKE_TEST_IMAGE || true

docker run -d --rm \
    --name $SMOKE_TEST_IMAGE \
    -e ASPNETCORE_ENVIRONMENT=Testing \
    $APP_IMAGE

# wait for the port to be available 
until nc -z $(docker inspect --format='{{.NetworkSettings.IPAddress}}' $SMOKE_TEST_IMAGE) $TEST_PORT
do
    echo "waiting for container to startup..."
    sleep 1.0
    ((MAX_WAIT_SECONDS--)) #decrement
    if [ $MAX_WAIT_SECONDS -le 0 ]; then
        echo "Docker smoke test failed, port $TEST_PORT not available"
        docker kill $SMOKE_TEST_IMAGE || true
        exit -1
    fi
done

# hit the status endpoint (don't kill script on failure)
set +e
docker exec -i $SMOKE_TEST_IMAGE wget http://localhost:$TEST_PORT$$TEST_PATH

# capture the return code
result=$?

# Reenable exit on error
set -e

# kill the test container
docker kill $SMOKE_TEST_IMAGE

if [ $result -ne 0 ]; then
    echo "Docker smoke test failed"
else
    echo "Docker smoke test passed"
fi

exit $result

To try and make it more manageable, I'll walk through the script, and describe the purpose of each section.

Walking through the script

The first section of the script is some standard boilerplate, and general configuration:

#!/bin/bash

# treat undefined variables as an error, and exit script immediately on error
set -eux

APP_NAME="$1"
APP_IMAGE="$2"
TEST_PATH="$3"
SMOKE_TEST_IMAGE="$APP_NAME-smoke-test"

TEST_PORT=80
MAX_WAIT_SECONDS=30

# kill any lingering test containers
docker kill $SMOKE_TEST_IMAGE || true

In this first section we read the three arguments passed to the script (see Running the Bash script below) using "$1", "$2", and "$3", and assign those to variables. We also define two constants TEST_PORT and MAX_WAIT_SECONDS that we'll use later.

Note: If you're new to Bash, make sure you don't put a space next to = when setting a variable name. Writing BASE_IMAGE = "$1" (with spaces) will give you errors!

Finally, we docker kill any lingering smoke test containers on the build agent. This shouldn't be necessary, as we'll cleanup the containers at the end of the script too, but it can occasionally happen if you have an error in your Bash script that prevents the cleanup from happening.

In the next section, we run our app in a container:

docker run -d --rm \
    --name $SMOKE_TEST_IMAGE \
    -e ASPNETCORE_ENVIRONMENT=Testing \
    $APP_IMAGE

I'll break down each line of this command (as you can't put comments on multiline commands):

  • docker run -d --rm - Docker will run the image in the background (-d). When the image exits (or is killed), the container will be removed (--rm).
  • --name $SMOKE_TEST_IMAGE - Use the variable $SMOKE_TEST_IMAGE as the name of the container. If the name of the app is testapp, then $SMOKE_TEST_IMAGE="testapp-smoke-test".
  • -e ASPNETCORE_ENVIRONMENT=Testing - I like to use a specific "Testing" environment for running smoke tests and integration tests, but you could alternatively use Development.
  • $APP_IMAGE - The Docker image of the app to run.

Note that I use the Testing (or Development) environment for the smoke test. You could run a smoke test for each environment that you're deploying to, to test the configuration in each of those environments. That would verify the app can startup in every environment. However I'm a little nervous about running apps using Production settings as part of testing, in case a bug (or feature) has inadvertent consequences. This is just a smoke test after all, it's not intended to be foolproof!

At this point, our app is running a container, on the CI server, listening on the default ports inside the container. However, as we haven't mapped any ports externally (using -p), there's no way for anything to call the app directly. This is intentional - we don't want the CI server randomly exposing the smoke test apps. Instead, we will use docker exec to probe the container shortly.

Before we try and exec into the container, we need to make sure it's started. In the following script we use docker inspect to look for evidence that the container is listening on our test port. If it isn't, the script sleeps for 1 second and tries again, until $MAX_WAIT_SECONDS seconds expires, at which point we accept that the smoke test failed, and exit the script.

until nc -z $(docker inspect --format='{{.NetworkSettings.IPAddress}}' $SMOKE_TEST_IMAGE) $TEST_PORT
do
    echo "waiting for container to startup..."
    sleep 1.0
    ((MAX_WAIT_SECONDS--)) #decrement
    if [ $MAX_WAIT_SECONDS -le 0 ]; then
        echo "Docker smoke test failed, port $TEST_PORT not available"
        docker kill $SMOKE_TEST_IMAGE || true
        exit -1
    fi
done

The next section of the script is where we test if the app health check endpoint is responding correctly using docker exec:

# hit the status endpoint (don't kill script on failure)
set +e
docker exec -i $SMOKE_TEST_IMAGE wget http://localhost:$TEST_PORT$$TEST_PATH

# capture the return code
result=$?

The first thing we do is set +e so that if the docker exec command fails, the script doesn't exit immediately. Instead, we capture the return code in the variable $result. The command we're running is a simple GET request for the provided test port and path. At runtime, variable replacement will mean this command looks something like;

docker exec -i testapp-smoke-test wget http://localhost:80/healthz

As long as the app returns a 200 OK or other success response, the smoke test will pass.

You could also use curl instead of wget. I use wget as it's available by default in the tiny .NET Core alpine Docker images.

The final part of the script is some simple cleanup, reenabling exit on error, killing our running smoke test container, and returning the smoke test result stored in $result

# Reenable exit on error
set -e

# kill the test container
docker kill $SMOKE_TEST_IMAGE

if [ $result -ne 0 ]; then
    echo "Docker smoke test failed"
else
    echo "Docker smoke test passed"
fi

exit $result

To use this script, add it to your repository somewhere, and invoke it as part of your build process, as shown in the following section.

Running the Bash script

To run the smoke test, first save the script to docker_smoke_test.sh, and invoke it passing in the name of the app, the docker image to run, and the path to invoke. If you're not using the default port 80 for your app, then you could customise the script to also pass that in as an argument.

./docker_smoke_test.sh "testapp" "my_private_repo/testapp" "/healthz"

Tip When you create the script, make sure you use only LF for the line endings (instead of the default CRLF on Windows). Also make sure to mark the script as executable in git.

After running the script, you should see something like the following output from your CI logs:

+ APP_NAME=testapp
+ APP_IMAGE=my_private_repo/testapp:latest
+ TEST_PATH=/healthz
+ APP_TEST_PORT=80
+ MAX_WAIT_SECONDS=30
+ docker kill testapp-smoke-test
Error response from daemon: Cannot kill container testapp-smoke-test: No such container: testapp-smoke-test
+ true
+ docker run -d --rm --name testapp-smoke-test -e ASPNETCORE_ENVIRONMENT=Testing my_private_repo/testapp:latest
++ docker inspect '--format={{.NetworkSettings.IPAddress}}' testapp-smoke-test
+ nc -z 172.17.0.8 80
+ echo 'waiting for container to startup...'
waiting for container to startup...
+ sleep 1.0
+ (( MAX_WAIT_SECONDS-- ))
+ '[' 29 -le 0 ']'
++ docker inspect '--format={{.NetworkSettings.IPAddress}}' testapp-smoke-test
+ nc -z 172.17.0.8 80
+ set +e
+ docker exec -i testapp-smoke-test wget http://localhost:80/healthz
Connecting to localhost:80 (127.0.0.1:80)
healthz               100% |*******************************|  6350   0:00:00 ETA

+ result=0
+ set -e
+ docker kill testapp-smoke-test
testapp-smoke-test
+ killOk=0
+ '[' 0 -ne 0 ']'
+ '[' 0 -ne 0 ']'
+ echo 'Docker smoke test passed'
Docker smoke test passed
+ exit 0

The important point here is the "Docker smoke test passed" printed to the console and exit 0, indicating that the smoke test passed successfully.

If you want a bit less verbosity, use set -eu instead of set -eux at the top of your test script, and commands will no longer be echoed to the console as they're executed.

Summary

In this post I showed how you could use Docker to run a smoke test as part of your CI build process. You can use this test to check that your application can start up and can respond to a basic endpoint. This is far from a thorough test, and doesn't cover as much as functional/integration tests would, but it can be useful nonetheless.