Adding Integration Tests to TFS Build Workflow

Overview

In my last post I described how to deploy web applications to a build integration server using Team Foundation Server 2010. The next logical step once the build is successfully deploying to the integration server is to trigger a set of integration tests to verify the deployment. In this post I will describe the changes to the Default Template build workflow to execute Integration Tests separately from the existing Unit Tests.

Unit Tests

It is important to consider at this stage why we would run integration unit tests, as opposed to the unit tests executed as part of the assembly build process.

Unit tests executed as part of the build are intended to verify the individual components are functioning correctly, and often would use mocked interfaces to ensure that only the specific functions being tested are executed. Unit tests are typically not reliant on deployed components and therefore can be run as soon as the assemblies have been built.

Integration tests on the other hand are intended to run against the fully deployed environment to ensure that the individual components successfully execute together. Integration tests therefore need to be executed after the application components have been deployed to an integration server. Failures in integration testing might indicate breaking changes such as database changes, missing data, or changed interfaces into other components of the system.

Note that running the deployment and integration tests adds to the duration required to execute a built. Rather than performing this action every time something in the solution changes it might be more pragmatic to have one Build Definition to build and run unit tests on a per-check-in basis, while another is configured for the full integration tests on a nightly basis.

Modify the Build Workflow

Workflow Sequence Overview

The integration tests have to run within the context of a build agent, to the activity needs to take place at the end of the Run On Agent activity, directly after the packages have been deployed to the build integration server within the Deploy Packages activity.

Changing variable scopes

Because we are going to borrow heavily from the existing “Run Tests” activity, but the execution will be outside the “Try Compile, Test, and Associate Changesets and Work Items” activity, we need to modify the scoping of the following variables. This is easiest done by editing the xaml directly in your favourite xml editor.

• outputDirectory – copy from the “Compile and Test for Configuration” activity up a level to the “Run On Agent” activity.
• treatTestFailureAsBuildFailure – copy from the try block of “Try Compile, Test, and Associate Changesets and Work Items” to the “Run On Agent” activity.

Add new Integration Tests workflow arguments

The parameters being added are as follows:

• Integration Tests Disabled (Boolean). I’m not a fan of negative argument types (eg, Disabled, rather than Enabled), however have decided to keep this consistent with the existing Tests Disabled argument.
• Integration Test Specs (TestSpecList).

The default value for the Integration Test Specs argument provides the defaults for filtering the unit tests to only the integration tests. Ideally I would have liked to be able to filter this to *test*.dll with a test category of Integration, however based on some rudimentary experimentation it appears that the Test Assembly Spec constructor can only set the assembly name filter. In the end I’ve used the following TestSpecList definition as the default value:

New Microsoft.TeamFoundation.Build.Workflow.Activities.TestSpecList(

New Microsoft.TeamFoundation.Build.Workflow.Activities.TestAssemblySpec

(“**\*test*.dll”))

Note: Don’t forget to change the Metadata property to ensure the new arguments are displayed in a suitable category in the Build Definition editor.

Add the Run Integration Tests Activity

Follow the following steps to add the new Run Integration Tests activity to the workflow

1. Add a new foreach activity after the Deploy Packages activity, but still within the Run on Agent activity. This activity will be used to iterate through the project configurations defined in the build definition.
   <ForEach x:TypeArguments=“mtbwa:PlatformConfiguration“ DisplayName=“Run Integration Tests“Values=“[BuildSettings.PlatformConfigurations]“>
   <ActivityAction x:TypeArguments=“mtbwa:PlatformConfiguration“>
   <ActivityAction.Argument>
   <DelegateInArgument x:TypeArguments=“mtbwa:PlatformConfiguration“Name=“platformConfiguration“ />
   </ActivityAction.Argument>
   …
   </ForEach>
2. Create a copy of the existing activity titled “If Not Disable Tests” into the foreach statement created above
3. Modify the copied workflow to use the added workflow arguments
   • Use Integration Tests Disabled instead of Disable Tests
   • Use Integration Test Specs instead of Test Specs

Configure the Build Definition

Configuring the filters for your integration tests is a matter for personal preference, though I’ve found the following approaches fairly simple;

• Define all integration tests in a separate project and utilise the Test Assembly Filespecfilter
• Add a Test Category of Integration to each of the tests and use the Category Filter.
• Configure a custom testsettings file to allow for accurately specifying the order tests should be executed

What’s Next?

Having the integration tests successfully executed is all fine and good however you will find that it is necessary to configure the endpoints in the app.config file of your unit tests project to always point to the integration server, which causes some inconvenience if you wish to run the same tests locally on a development environment.

In a future post, I will have a look at how to perform transformations on the app.config file as part of the deployment, similar to the way web.config is transformed as part of the deployment package creation.